The Liberal Democrats had held the seat from 1997 until 2015, when it was won by Davies. In the 2017 snap election he held the seat with a majority of just over 8,000 votes.
Johnson, who took office last week, has said he does not plan to hold an election before Britain leaves the EU but could be forced to if lawmakers try to stop him pursuing a no-deal exit by collapsing the government in a no confidence vote.
Some Conservative lawmakers have hinted they would vote against the government to thwart a no-deal Brexit. Party chairman James Cleverly said it was incumbent upon them to support Johnson.
"This is the long-standing Conservative Party policy, the party policy we were all elected on which is to seek a deal from the European Union, but if that is not forthcoming, then to leave the European Union whatever," Cleverly told BBC radio.
The result in Brecon, where the combined vote for pro-leave parties outnumbered pro-remain, would likely add to calls for a possible Conservative-Brexit Party alliance in any election.
Pro-EU parties, including the Greens and Plaid Cymru, agreed not to stand in the Brecon election on Thursday to boost the Liberal Democrats chances by concentrating the support of 'Remain' voters behind one candidate.
Swinson said it was likely the parties would seek similar agreements in any future parliamentary election.
The Brexit Party, led by prominent Brexit campaigner Nigel Farage, was only launched in April and rode a wave of anger over the government's failure to deliver Brexit to sweep to victory in the United Kingdom's European Parliament election in May.
