Opinion for the court filed by Circuit Judge BRYSON. Opinion dissenting-in-part filed by Circuit Judge CLEVENGER.

Roger Tudor appeals from a decision of the Merit Systems Protection Board sustaining his demotion by the Internal Revenue Service from Supervisory Special Agent to Investigative Analyst. Because the administrative judge misconstrued an important component of the evidence in the case, we reverse the Board’s decision and remand the case to the Board for further proceedings.

I

Beginning in 2003, Mr. Tudor was employed as a Supervisory Special Agent with the IRS Criminal Investigation office in Dallas, Texas. In that capacity, he directed a group of IRS Special Agents investigating criminal tax fraud. His group engaged in non-traditional IRS activity, whereby investigations would take place very rapidly in response to bulk currency and money laundering activities discovered in the field.

Mr. Tudor was directly supervised by an Assistant Special Agent in Charge (“ASAC”) in the Dallas field office. Georgia Taylor was the ASAC responsible for Mr. Tudor’s supervision for most of the time period at issue in this case. For two months in late 2004 and for four months in mid-2005, Ms. Taylor left the Dallas office on assignment and was replaced by Catherine Tucker as Acting ASAC. The ASAC or Acting ASAC was supervised by Special Agent in Charge (“SAC”) Michael Lacenski. The SAC was, in turn, supervised by the Dallas Director of Field Operations (“DFO”), John Imhoff. After the period at issue in this case, SAC Lacenski and DFO Imhoff were succeeded by Erick Martinez and Michael Lahey, respectively.

In late 2005, DFO Lahey ordered an IRS Management Inquiry in response to allegations that Mr. Tudor had manipulated data entered into the agency’s Criminal Information Management Inventory System (“CIMIS”). The Management Inquiry was conducted by two IRS employees, Debra King and Charles Pearre. In the course of the inquiry, Ms. King and Mr. Pearre interviewed Mr. Tudor and Mr. Lacenski, preparing memoranda based on those interviews. Ultimately, Ms. King and Mr. Pearre concluded that Mr. Tudor had manipulated information entered in CIMIS and had circumvented IRS approval policies on several occasions between February 2004 and July 2005. The Treasury Inspector General for Tax Administration followed up on the Management Inquiry with its own investigation. As part of that investigation, Treasury Department special agents conducted their own interview of Mr. Tudor. The outcome of the Inspector General’s investigation was a November 2006 report that substantiated seven allegations of misconduct on Mr. Tudor’s part.

The Inspector General’s report was referred to the IRS for resolution. In June 2007, SAC Martinez issued a notice to Mr. Tudor proposing to demote him to Investigative Analyst based on three charges: (1) failure to obtain appropriate approval authority for certain actions; (2) unauthorized disclosure of taxpayer information; and (3) repeated entry of false information in CIMIS. The first charge consisted of 64 specifications: 35 specifications of improper initiation of investigations without authority; 28 specifications of improper referral of IRS investigations to U.S. Attorneys’ offices for prosecution; and one specification of improper initiation of a *1364subject seizure investigation. For each specification, Mr. Tudor was alleged to have improperly signed a Criminal Investigation General/Primary/Subject Investigation Report, known as Form 4930. In November 2007, deciding official Steven Pregozen issued a notice of demotion sustaining all of the charges and specifications.

Mr. Tudor appealed his demotion to the Merit Systems Protection Board. The administrative judge who was assigned to the case sustained the IRS action based on the 28 referral specifications of the first charge and the second charge of unauthorized disclosure of taxpayer information. The administrative judge did not make conclusive determinations regarding the remaining 36 specifications of the first charge, and he explicitly found that the third charge of entry of false information was unsubstantiated. Because Mr. Pregozen testified that he would have imposed the same penalty on Mr. Tudor based on the first charge alone, the administrative judge found the penalty of demotion appropriate. Mr. Tudor appealed to the full Board, which had two members at the time. The Board could not agree on the proper disposition of the case, so the decision of the administrative judge became the final decision of the Board. This appeal followed.

II

The sole issue presented on appeal is whether there is adequate evidentiary support for the administrative judge’s findings related to the 28 specifications of the first charge dealing with improper referrals of investigations to U.S. Attorneys’ offices for -prosecution. Mr. Tudor argues that his uncontradicted testimony before the administrative judge established that he was given verbal authorization by his superiors to directly refer investigations to U.S. Attorneys’ offices. The Treasury Department responds that Mr. Tudor’s referral activities were undisputedly contrary to IRS policy, that the administrative judge’s finding that Mr. Tudor did not have verbal authorization to refer investigations was based on a virtually unreviewable credibility determination, and that the record as a whole supports the administrative judge’s conclusion.

The Treasury Department is correct that IRS policy prohibited Mr. Tudor from referring cases to U.S. Attorneys’ offices for prosecution. See Internal Revenue Manual §§ 9.5.10.4-.5 (2002) (requiring SAC to make prosecution referrals). Mr. Tudor admitted as much in his testimony before the administrative judge, and he does not challenge that determination here. However, according to the deciding official, Mr. Tudor’s noncompliance with IRS policy was not in itself sufficient to sustain the charge of failure to obtain appropriate approval authority. Mr. Pregozen testified that he would have not sustained the first charge if Mr. Tudor’s superiors had condoned his approval activities, even though those activities were contrary to IRS policy. Mr. Tudor challenges the administrative judge’s conclusion that it was “inherently improbable” that either the SAC or the ASAC granted Mr. Tudor approval to refer investigations for prosecution.

Mr. Tudor testified at the hearing before the administrative judge that he believed he possessed “assumed, implied authority” from SAC Laeenski to refer investigations for prosecution. According to Mr. Tudor, SAC Laeenski “would give [him] verbal approval” to initiate investigations and refer investigations to U.S. Attorneys’ offices. Mr. Tudor stated that SAC Laeenski had given him “a lot of leeway” to depart from IRS approval policies. He added that he “went ahead *1365and processed the case through like [SAC Laeenski] instructed,” in the belief that his actions were proper. Mr. Tudor testified that he discussed cases that he referred for prosecution with his SAC “several times” and with his ASAC over the telephone or during routine workload reviews. He stated that he was never reprimanded by SAC Laeenski, ASAC Taylor, Acting ASAC Tucker, or anyone else for directly referring investigations to U.S. Attorneys’ offices, and that he would have stopped referring investigations if he had been told not to do so.

The administrative judge did not credit Mr. Tudor’s testimony because he regarded it as inconsistent with the testimony of SAC Laeenski. The administrative judge found SAC Laeenski to be “very clear that he had never authorized [Mr. Tudor] to refer any cases to the U.S. Attorney.” The administrative judge’s decision to credit SAC Lacenski’s testimony over that of Mr. Tudor on this critical issue would be permissible if the two were in conflict, as the administrative judge believed. However, the administrative judge’s characterization of SAC Lacenski’s testimony is unsupported by the hearing transcript. The sole reference in SAC Lacenski’s testimony to referrals for prosecution, as opposed to investigation initiations, was the following sentence:

When cases were referred for prosecution, my instructions to both Mr. Tudor and the Assistant Special Agent in Charge was that that would have to be a situation that was discussed with the Assistant Special Agent in Charge.

That statement is not inconsistent with Mr. Tudor’s testimony that he was given verbal approval authority by his superiors to refer investigations for prosecution, and that he discussed those referred investigations with his ASAC.

SAC Laeenski never testified that he did not give approval authority for Mr. Tudor to refer investigations for prosecution. In the statement on which the administrative judge may have relied, SAC Laeenski stated that “the extent of [his] approval for Mr. Tudor to initiate cases” was limited to a small number of cases (around five) dealing with a bulk currency initiative. That testimony, however, dealt with initiation of investigations, not referral of cases to U.S. Attorneys’ offices. The administrative judge did not rely on evidence that the limitations on Mr. Tudor’s authority to initiate investigations extended to his authority to refer cases for prosecution. The administrative judge’s finding that “[SAC] Laeenski specifically stated ... that he never authorized [Mr. Tudor] to make referrals to the U.S. Attorney” is therefore unsupported by the hearing transcript and cannot form the basis for discrediting Mr. Tudor’s testimony to the contrary. The great deference owed to an administrative judge’s credibility determination in resolving conflicting testimony, see Bieber v. Dep’t of the Army, 287 F.3d 1358, 1364 (Fed.Cir. 2002), is irrelevant if the testimony in question is not in conflict.

The administrative judge also found Mr. Tudor’s claim of authority to refer investigations to U.S. Attorneys’ offices “at complete odds with ... the testimony of every other witness.” Those other witnesses were Acting ASAC Tucker, Ms. King, and IRS Special Agent Mike Parsons. The scope of Mr. Parsons’ testimony was limited to the single specification of the first charge for improper approval of a subject seizure investigation, so it is not relevant to the issue on appeal. The testimony of Acting ASAC Tucker did not shed light on the question of whether SAC Laeenski or ASAC Taylor had given Mr. Tudor verbal authority to refer investigations for prosecution. Ms. Tucker stated that she “did *1366not review” policies for delegation of approval authority in the Dallas field office. She did, however, recognize that the approval process for initiation of investigations in the office deviated from IRS policy, as reflected in the Internal Revenue Manual; whereas the manual required investigations to be initiated at the SAC level, the Dallas local rules allowed the ASAC to initiate investigations.

The remaining witness, Ms. King, testified that in the course of her Management Inquiry she did not discover documents showing that Mr. Tudor was delegated approval authority to refer investigations for prosecution. However, as the administrative judge noted, Ms. King “did not make any finding regarding the delegation of authority because she did not see it as her task.” While stating that ASAC Taylor “didn’t recall” giving Mr. Tudor approval authority, Ms. King explained that she did not inquire as to whether Mr. Tudor’s approval activities were known to ASAC Taylor. Like Acting ASAC Tucker, Ms. King admitted in her testimony that the Dallas field office had deviated from IRS policy regarding delegation of approval authority. Ms. King also thought she remembered SAC Lacenski telling her that DFO Imhoff had given verbal permission for the Dallas field office to deviate from approval policies in the Internal Revenue Manual.

In sum, none of the other witnesses — SAC Lacenski, Acting ASAC Tucker, Ms. King, or Mr. Parsons — contradicted Mr. Tudor’s testimony that he had been given oral authority to refer investigations to U.S. Attorneys’ offices for prosecution. Indeed, given the apparent verbal instruction by DFO Imhoff to deviate from IRS policy in delegating approval authority, and SAC Lacenski’s admission that he had improperly delegated at least some approval authority to Mr. Tudor, it does not seem “inherently improbable” that Mr. Tudor was granted verbal authority to directly refer investigations for prosecution in consultation with his ASAC. We must set aside findings when they are unsupported by substantial evidence, 5 U.S.C. § 7703(c)(3), and “[t]he substantiality of evidence must take into account whatever in the record fairly detracts from its weight,” Universal Camera Corp. v. NLRB, 340 U.S. 474, 488, 71 S.Ct. 456, 95 L.Ed. 456 (1951); accord, Leatherbury v. Dep’t of the Army, 524 F.3d 1293, 1300 (Fed.Cir. 2008). The first charge of failure to obtain appropriate approval authority must be set aside because the administrative judge did not support his finding with substantial evidence.

Mr. Pregozen, the deciding official, testified that the second charge of unauthorized disclosure of taxpayer information would “probably not” merit the same penalty of demotion were it to stand alone. Therefore, we vacate the Board’s decision and remand for a determination of the appropriate penalty for Mr. Tudor.

We do not decide whether any of the 64 specifications of the first charge could be supported by evidence in the administrative record that the administrative judge did not address in his initial decision; instead, we leave that issue to be addressed by the Board on remand. In the remand proceedings, the Board is free to review all of the evidence in order to determine whether other evidence in the record, not affected by the administrative judge’s misapprehension as to the import of SAC Lacenski’s testimony, justifies a conclusion that the agency has met its burden of proof with respect to the first charge. The task of assessing that evidence is for the finder of fact in the first instance, and in light of the conflicts in the evidence and the inferences that need to be drawn from some of that evidence, we conclude that *1367the best course is to remand the case for the Board’s reconsideration, free of the characterization of SAC Lacenski’s testimony that affected the initial decision in this case. See Vidal v. U.S. Postal Serv., 143 F.3d 1475, 1480 (Fed.Cir. 1998); Holmes v. Dep’t of Veterans Affairs, 58 F.3d 628, 633 (Fed.Cir. 1995).

Costs to Mr. Tudor.

VACATED AND REMANDED

dissenting-in-part.

I respectfully dissent from the majority’s decision to expose Mr. Tudor on remand to further litigation regarding the first charge. As the majority amply demonstrates, Mr. Tudor was acting on his own when initiating investigations or referring investigations for prosecution. However, whether or not Mr. Tudor had actual authority to do so, Mr. Pregozen, the deciding official, testified that he would not have sustained the first charge if Mr. Tudor’s supervisors had condoned his case initiations and referrals, which were reflected in the 4930 forms he signed. Black’s Law Dictionary defines “condone” as “to voluntarily ... overlook.” Black’s Law Dictionary 336 (9th ed. 2009). Merriam-Webster likewise equates overlooking with condoning. See Merriam-Webster’s Online Thesaurus, http://merriamwebster.com/thesaurus/condone (last visited Apr. 19, 2011). Among the many synonyms for “condone” are: “countenance,” “let pass,” “yield or submit to,” “hear of and go along with,” “look the other way,” and “be content with.” See Roget’s International Thesaurus §§ 134.7, 443.10, 978.7 (6th ed. 2001).

The record reveals that DFO Imhoff and Mr. Lacenski gave Mr. Tudor at least some authority to act on his own initiative in consultation with Ms. Taylor. Because Mr. Tudor had no formal approval to sign the 4930 forms, when those forms were annually reviewed by the Dallas office, Mr. Tudor’s activities necessarily came to light, as Mr. Lacenski testified. Moreover, as Mr. Lacenski further testified, despite the numerous occasions on which the Dallas office and headquarters reviewed the 4930 forms that Mr. Tudor signed, nobody voiced any concern about Mr. Lacenski’s delegation of authority to Mr. Tudor. Mr. Tudor testified that both Mr. Lacenski and Ms. Taylor condoned his approval activities. Mr. Lacenski did not contradict Mr. Tudor’s testimony and the agency did not call Ms. Taylor as a witness. Mr. Lacenski testified that he personally did not review the 4930 forms that Mr. Tudor filed, but he testified that Mr. Tudor and Ms. Taylor periodically briefed him about cases that were initiated and prosecuted. Neither the Dallas office nor anyone else reprimanded or took any action to stop Mr. Tudor from what he was doing for a long period of time. Although the precise word “condone” was not used in testimony to describe the reaction of Mr. Tudor’s supervisors to his approval activities, the extensive testimony on the subject taken as a whole proves that the supervisors in fact condoned the conduct covered by the first charge.

Even if there are inconsistencies between Mr. Tudor’s trial testimony and statements he allegedly made to investigators as to whether or how much authority he had been given to initiate or refer cases, there is no challenge to Mr. Tudor’s claim that his supervisors condoned his activities.

We should not lose sight of the fact that Mr. Tudor got caught in a trap set for him by his supervisors. Due to the fast-paced and pressing nature of the cases on which Mr. Tudor worked, his supervisors decided, contrary to the “book,” to look the other way and let him initiate and refer *1368cases, and sign a form when he had no written authority to do so. By every light shown on the record, Mr. Tudor did a fíne job, and for a time must have pleased his supervisors. The fly in the ointment came when Ms. Tucker replaced Ms. Taylor as acting ASAC and Mr. Tudor’s activities became known beyond the ImhoffiLacenski/Taylor orbit. Not surprisingly, a major investigation immediately ensued, with Mr. Tudor as the fall guy. In the end, the deciding official got it just right: even though the agency probably should have made sure Mr. Tudor had no chance to sign the 4930 forms, he should not be punished if his supervisors condoned his activities.

Because Mr. Tudor’s supervisors condoned the activities covered by the first charge, Mr. Tudor cannot be punished under that charge. Thus, the proper course is not to remand the case for further review of the merits of the first charge but, rather, to remand the case to let the agency decide the penalty it wishes to assess for the sustained second charge of unauthorized disclosure of taxpayer information. Mr. Tudor should not be exposed to further litigation on the first charge.

For these reasons, I respectfully dissent, although I too would remand the case, but for a quite different purpose.