Impact of the HIV-1 genetic background and HIV-1 population size on the evolution of raltegravir resistance
© The Author(s) 2018
Received: 18 September 2017
Accepted: 23 December 2017
Published: 5 January 2018
Emergence of resistance against integrase inhibitor raltegravir in human immunodeficiency virus type 1 (HIV-1) patients is generally associated with selection of one of three signature mutations: Y143C/R, Q148K/H/R or N155H, representing three distinct resistance pathways. The mechanisms that drive selection of a specific pathway are still poorly understood. We investigated the impact of the HIV-1 genetic background and population dynamics on the emergence of raltegravir resistance. Using deep sequencing we analyzed the integrase coding sequence (CDS) in longitudinal samples from five patients who initiated raltegravir plus optimized background therapy at viral loads > 5000 copies/ml. To investigate the role of the HIV-1 genetic background we created recombinant viruses containing the viral integrase coding region from pre-raltegravir samples from two patients in whom raltegravir resistance developed through different pathways. The in vitro selections performed with these recombinant viruses were designed to mimic natural population bottlenecks.
Deep sequencing analysis of the viral integrase CDS revealed that the virological response to raltegravir containing therapy inversely correlated with the relative amount of unique sequence variants that emerged suggesting diversifying selection during drug pressure. In 4/5 patients multiple signature mutations representing different resistance pathways were observed. Interestingly, the resistant population can consist of a single resistant variant that completely dominates the population but also of multiple variants from different resistance pathways that coexist in the viral population. We also found evidence for increased diversification after stronger bottlenecks. In vitro selections with low viral titers, mimicking population bottlenecks, revealed that both recombinant viruses and HXB2 reference virus were able to select mutations from different resistance pathways, although typically only one resistance pathway emerged in each individual culture.
The generation of a specific raltegravir resistant variant is not predisposed in the genetic background of the viral integrase CDS. Typically, in the early phases of therapy failure the sequence space is explored and multiple resistance pathways emerge and then compete for dominance which frequently results in a switch of the dominant population over time towards the fittest variant or even multiple variants of similar fitness that can coexist in the viral population.
Currently, viral replication is successfully suppressed in the majority of HIV-infected patients treated with combination antiretroviral therapy (cART) . However, virological failure associated with the emergence of drug resistant viruses may still limit the success of cART. The emergence of drug resistance in HIV is a direct consequence of the high error-rate of the HIV reverse transcriptase (RT) enzyme [2–4]. The frequent incorrect nucleotide incorporations result in evolution of the viral population and generate a myriad of viral variants upon which selective forces may act. The population size and replication rate are important viral parameters that contribute to the probability that resistance emerges and to how HIV-1 drug resistance evolves [5–9].
Integrase inhibitors comprise a class of antiretroviral drugs that specifically prevent the integration of the viral genome into the human genome. Raltegravir is the first representative of a class of integrase inhibitors that target the strand transfer reaction (INSTIs) of the viral DNA into the host genome which is performed by the viral enzyme integrase. Like other INSTIs, raltegravir preferentially binds and inhibits the viral DNA-integrase complex (intasome) over unbound integrase [10–12].
It was the first integrase inhibitor used in clinical practice (since 2007) but was recently registered by the FDA for once daily dosing  and is very well tolerated  due to a low toxicity profile . Resistance is commonly associated with selection of one of the signature raltegravir resistance mutations Y143C/R/H, Q148H/K/R or N155H [16–18]. Mutations at each of these three amino acids represent a distinct resistance pathway and all signature mutations are associated with reductions in viral replication [17, 19, 20]. Accumulation of secondary resistance mutations is often associated with a greater loss of drug susceptibility and/or improved viral fitness [21–23]. Different mutational combinations vary greatly in their impact on raltegravir susceptibility and viral replication. In general, substitutions at amino acid position 148 confer higher levels of resistance than substitutions at amino acid Y143 or N155. The G140S plus Q148H combination is considered the most resistant variant and has little effect on viral replication. The resistance patterns observed in HIV-1 patients on a raltegravir containing regimen are very diverse and Q148H/K/R (usually with G140A/C/S and/or E138A/K) and N155H (often together with E92Q or V151I) mutations are observed more frequently than Y143 mutations . The different resistance pathways are believed to be mutually exclusive and multiple primary mutations (especially 148 + 155 mutations) are generally not observed on the same viral genome . Remarkably, replacement of the dominant resistant population by a viral population with a completely different resistance pattern during continuous non-suppressive INSTI therapy has been observed [26–29].
The mechanisms that drive selection and switching of resistance pathways are inadequately understood. Understanding these mechanisms is essential in view of other INSTIs that are in clinical use (elvitegravir and dolutegravir) or in clinical trial (bictegravir and cabotegravir), since their resistance profiles partially overlap with that of raltegravir . For instance, raltegravir resistance mutations Q148H/K/R and N155H show a high level of cross-resistance with elvitegravir, but elvitegravir susceptibility is unaffected by Y143 mutations [30, 31], a difference that is beautifully explained by crystal structures of the intasome in presence of raltegravir or elvitegravir . The VIKING studies demonstrated dolutegravir’s superior resistance profile attested by sustained activity against all raltegravir resistant variants, except for viruses with a mutation at amino acid 148 in combination with at least one secondary mutation at position 138 and/or 140 [32–34]. These observations were corroborated by in vitro analysis of resistance profiles [35–37] which also uncovered two atypical INSTI resistance mutations (G118R and F121Y) that confer pan-INSTI resistance [38, 39].
We investigated the impact of the HIV-1 genetic background and population size on the evolution of raltegravir resistance and their role in determining selection of a particular resistance pathway. Using next-generation sequencing (NGS) we analyzed patient-derived viral integrase sequences from samples taken before and during raltegravir therapy failure. Frequency and dynamics analysis of the deep sequencing data was used to evaluate intra-patient evolution of resistance.
To investigate the role of the genetic background we generated recombinant viruses containing the viral integrase CDS from pre-raltegravir samples from two patients experiencing virological failure receiving raltegravir therapy. With these recombinant viruses we conducted multiple low multiplicity of infection (MOI) in vitro selections in parallel, with the advantage that all resistant variants generated are allowed to replicate and different raltegravir resistance pathways can be identified.
During raltegravir resistance development, multiple resistant variants emerge that compete to become the dominant variant
We studied five patients who initiated raltegravir therapy as part of a cART regimen and subsequently demonstrated virological failure to the raltegravir containing regimen. Of note, all 5 patients were heavily pre-treated and raltegravir was part of a (partly) compromised backbone. This may partially explain the moderate virological responses and limited therapy success in these patients. Patient (Pt) 1, Pt2, Pt3 and Pt5 were infected with HIV-1 subtype B strains, Pt4 was infected with a HIV-1 CRF02_AG strain. All patients started raltegravir therapy with viral loads > 5000 copies/ml (c/ml) of HIV-1 RNA, as measured in the last viral load test before initiation of raltegravir therapy. To investigate if the emerged resistant variants existed as minority variants before raltegravir therapy and how the resistant population evolved we analyzed longitudinal samples from these patients by NGS.
Pt2 was off therapy for a few weeks due to toxicity related issues but a new five-drug regimen including raltegravir and darunavir resulted in a rapid decline of the viral load but was discontinued again because of darunavir-related toxicity (Fig. 1b). Shortly thereafter, the same regimen was restarted without darunavir resulting in a further decline of the viral load to undetectable levels. After brief virological suppression (HIV-1 RNA < 50 c/ml, a 3.0 log decrease in HIV-1 RNA), viral load rebounded and population sequencing showed gradual accumulation of raltegravir resistance mutations: initially primary resistance mutation N155H appeared followed by two secondary resistance mutations, first Q95K and later V151I. NGS revealed no major raltegravir resistance mutations at baseline or mutations at positions 143 or 148 at any time-point, not even as minority variants (Fig. 2).
Pt3 only showed a partial virological response to the raltegravir containing therapy (a 1.0 log decrease in HIV-1 RNA was observed) and full suppression was never achieved. The viral load rebounded shortly after start of raltegravir containing cART (< 70 days, Fig. 1c) and the viral population contained N155H mutants. In a later time-point, 169 days after start of raltegravir, this population was replaced by a variant with primary mutation Y143R and several secondary mutations including L74M, T97A and G163E. NGS revealed only secondary mutations in the sample before raltegravir therapy, mostly as very low frequency variants (Fig. 2), including T97A and Y143C (both 0.2%). In a sample taken 28 days later, 77.5% of the population contained N155H, 15.4% had T97A and variants with Y143C were not detected anymore. In the subsequent sample 70 days after start of raltegravir therapy, N155H had increased to 88.6% and Y143C had reappeared in 11.8% of the reads. Y143C and N155H did not appear to be on the same genome (Fig. 2b). After 169 days the resistant population had shifted dramatically, with Y143R making up 99.7% of the population and complete absence of N155H. Mutations L74M, T97A and G163R appeared in nearly the entire viral population.
Pt4 was also a partial responder (1.1 log decrease in HIV-1 RNA) in whom virological suppression was not achieved during 48 weeks of raltegravir containing therapy. Sanger sequencing revealed only very small (< 20%) populations of Q148H/Q148R and N155H and resistant variants never dominated the population (Fig. 1d). Before raltegravir therapy, mutation Y143H was detected at low frequency (0.2%) by NGS but did not appear in any of the later samples during raltegravir treatment. In samples 8 and 81 days after start of raltegravir therapy also no significant raltegravir resistance mutations were observed (Fig. 2a). However, in the sample 123 days after raltegravir therapy initiation several primary and secondary raltegravir resistance mutations occurred. N155H was found in 14.4%, of the population, Q148H in 7.5% of which approximately half (4.1%) in combination with G140S and the remainder in combination with E157D (Fig. 2b). Q148R was present in 2.4% of the population and G163R in 2.7%.
In addition to a compromised backbone, this patient also had sub-therapeutic levels of raltegravir levels due to a drug–drug interaction with rifampicin (an interaction unknown at time of raltegravir prescription). Despite the suboptimal levels of raltegravir, switching to a raltegravir containing regimen quickly resulted in complete viral suppression (< 26 days, 3.3 log decrease in HIV-1 RNA) but viral load rebounded within 21 weeks (Fig. 1e). Population sequencing (sample 457 days after initiation of raltegravir) demonstrated that virus with integrase substitutions G140S + Q148H completely dominated the viral population.
No major resistance mutations were detected at baseline by NGS. Interestingly, in the first sample during therapy failure (blue nodes Fig. 1e) mutations from all three major resistance pathways including double mutants with secondary mutations were observed with G140S + Q148H being the most frequently occurring double mutant (Fig. 2). In this sample the variant with mutation N155H was the dominant variant but was outcompeted in the subsequent sample (red nodes Fig. 1e) by double mutant G140S + Q148H.
Evidence for elevated diversification following extinction from drug pressure on large viral populations
The genetic background is not paramount for the emerging INSTI resistance pathway
To investigate the role of the viral genetic background in determining the raltegravir resistance pathway we cloned the integrase coding region from pre-raltegravir therapy samples from two patients (Pt1 and Pt2) in an HXB2 reference background. These recombinant viruses were derived from amplicons of pre-raltegravir samples with a viral load of 21,800 and 51,600 c/ml respectively, creating libraries containing thousands of patient-derived sequences for both samples.
In these patients, raltegravir resistance developed through different pathways during raltegravir therapy. In Pt1, resistance developed initially through E138K + Q148K and later a second variant emerged with N155H (Fig. 1a). In Pt2, raltegravir resistance developed initially through N155H and was complemented by two secondary mutations, Q95K and V151I, which made up nearly 100% of the viral population during prolonged therapy failure (Fig. 1b). With the recombinant viruses of these raltegravir baseline samples and HXB2 reference virus (molecular clone from pHXB2AF, therefore a single sequence input per replicate) multiple in vitro selections were performed in parallel.
We investigated the impact of the genetic background and viral population size on the development and evolution of raltegravir resistance in vitro and in vivo. Deep sequencing revealed presence of major raltegravir resistance mutations at baseline in 3/5 patients (patients (1, 3 and 4, Fig. 2a) at very low frequencies (≤ 0.2%). Only the major resistance mutation detected before therapy in Pt3, Y143C (0.2% in 2nd sample on day 0), ultimately became the dominant variant in the resistant population although it surprisingly disappeared from the subsequent sample to reappear in the 4th sample (11.8% on day 70). In the final sample (169 days) Y143C was not detected anymore but virus with mutation Y143R completely dominated the viral population. In the majority of subtype B viruses, the Y143R substitution requires two nucleotide changes (Y143 = TAC → TGC = 143C → CGC = 143R). So it appears that the Y143C variant detected at baseline facilitated selection of the more resistant Y143R variant  in this patient. In Pt 1, secondary resistance mutation E138K was detected at baseline (0.1%) and appeared to have acquired Q148K as in the subsequent sample a double mutant with E138K + Q148K was detected (0.5%) which persevered in the population (44.7% in the final sample). This suggests that minor variants present at baseline can play a role in the development and evolution of raltegravir resistance but are not essential for the emergence of resistance [40–42], which is also seen for other drug classes [43, 44].
Remarkably, in Pt1 a variant with major mutation N155H and no apparent secondary resistance mutations emerged after and then co-existed alongside the E138K + Q148K double mutant, each comprising roughly 50% of the viral population. This suggests a fitness advantage for the N155H single mutant over the E138K + Q148K double mutant. This is unexpected considering the in vitro observations regarding resistance and replication of these mutants [17, 45, 46]. A possible explanation could be the so-called hitchhiking effect; this particular variant had a fitness advantage that was located outside the investigated region. For instance, resistance against any of the other antiretrovirals in the therapy regimen (e.g. efavirenz) could have been present in this variant but not in the other variants. While the chances that this occurs in one of the minority variants and not in any of the dominant species are small, the possibility cannot be excluded. Another possible explanation is that the variant with the N155H mutation is more fit than the E138K + Q148K mutant in the presence of raltegravir in this particular setting due to other factors (e.g. specific viral genetic background, immunological host factors, etc.).
We also observed a correlation between the number of unique sequence variants emerging and the magnitude of the virological response. Good virological response in Pts 1, 2 and 5 (> 2 log reductions in HIV-1 RNA) coincided with the number of unique sequence variants observed; the proportion of unique sequence variants increased when the viral load dropped. Pts 3 and 4 demonstrated moderate responses with viral load drops of around 1 log and showed no change in the relative frequencies of unique variants over the course of sampling. This suggests that in Pts 1, 2 and 5, treatment with raltegravir induced an elevated diversification to escape drug pressure while in Pts 3 and 4 the pressure on the population seemed to occur to a much lesser extent. This observation reminds of the explosive diversifications on a macro-evolutionary scale observed in other fields of biology after ice ages and the cataclysmic extinction of dinosaurs [47, 48]. However, regardless of treatment efficacy, i.e. both in patients with less dramatic and more severe virus load reductions, raltegravir resistance mutations developed and multiple resistance pathways were observed in 4/5 patients. The large extinction opens up previously occupied niches for new virus variants, (1) such that all new mutations are accepted and not compete for resources until the population regains a size limited by the carrying capacity of the system, or (2) most of the diversity pre-existed as a permanent, extremely low frequency pool of highly diverse viruses persisting in the shadow of more fit and high frequency variants. Once the high frequency variants are eradicated (e.g. by newly introduced drugs) a glimpse of that diversity surfaces. The larger the impact on the high frequency variants (i.e. reduction in viral load) the larger the proportion of the low frequency pool appears (i.e. number of unique variants increases when the viral load drops). Subsequently, one or two of the low frequency variants gain fitness in the new environment and become dominant, lowering the mean diversity again. The same can be argued from the point of viral escape. If the viral population doesn’t have to go that low to find fit variants (in the new environment), the impact on viral load is also less severe.
The raltegravir in vitro selections in which we mimicked population bottlenecking by using a low MOI allowing all resistant variants that are generated to emerge, clearly indicate that the resistance pathway that is selected to escape raltegravir pressure is not predisposed in the genetic background of the integrase CDS. Evaluation of the combined in vivo and in vitro data indicates that stochastic selection plays a major role during the initial development of raltegravir resistance.
In conclusion, the development and evolution of raltegravir resistance can be separated in multiple phases/components: (1) minority variants present at baseline can contribute to the emergence of raltegravir resistance but this is not preordained and seems to occur arbitrarily; (2) during the viral load drop due to drug pressure a burst of new sequence variants emerges creating diversifying selection; (3) these new variants usually include multiple raltegravir resistant variants (from multiple resistance pathways) that can pass the imposed bottleneck; (4) competition of these resistant variants determines the ultimate shape of the viral population. The resistant population can be the product of a single variant that outcompetes all others and only one variant represents the population or multiple variants with a similar fitness emerge that coexist in the viral population.
Further investigation is needed to better assess the exact impact of baseline minority resistance variants and the population size on the development and evolution of raltegravir resistance and determine the clinical implications of these factors.
Emergence of resistance against integrase inhibitor raltegravir in HIV-1 patients is generally associated with selection of one of three distinct resistance pathways. The mechanisms that drive selection of a specific pathway are still poorly understood. Using deep sequencing we observed an inverse correlation between the virological response and the relative amount of unique sequence variants emerging, suggesting diversifying selection during drug pressure. In 4/5 patients multiple signature mutations representing different resistance pathways were observed. In addition, in vitro selections revealed that identical viral clones were also able to select mutations from different resistance pathways indicating that raltegravir resistance is not predisposed in the genetic background of the viral integrase. Importantly, raltegravir resistance develops progressively and discontinuation during early phases of therapy failure is justified to preserve future options with second-generation INSTIs.
HIV-1 RNA was isolated using the Nuclisens Islolation kit (BioMérieux, Boxtel, The Netherlands). Briefly, 100 μl of sample was mixed with 900 μl lysisbuffer and 40 μl silica and incubated for 10 min at room temperature to allow binding of the nucleic acid to the silica particles. Unbound material was removed by several washing steps after which the RNA was eluted at 56 °C with 100 μl of 40 ng/µl poly-A RNA. The isolated viral RNA was used to reverse transcribe and amplify the viral integrase coding region in a single-step reaction using the Titan One Tube RT-PCR kit (Roche). The RT-PCR was conducted with primers 5′INoutF1 (5′-GGA ATC ATT CAA GCA CAA CCA GA-3′; 4059–4081) and 3′INoutR1 (5′-TGT ATG CAG ACC CCA ATA TGT TG-3′; 5262–5241). The amount of amplified product was further enhanced in a second PCR using the Expand High fidelity kit (Roche) with primers 5′INinF1 (5′-TAT CTG GCA TGG GTA CCA GCA C-3′; 4143–4164) and 3′INinR1 (5′-TAG TGG GAT GTG TAC TTC TGA AC-3′; 5217–5195). All PCR-amplified products were purified using the QIAquick PCR purification kit (Qiagen, Leusden, The Netherlands). Sequence analysis was performed with the BigDye Terminator v3.1 Cycle Sequencing Kit (Applied Biosystems, Foster City, CA, USA). Integrase sequences were obtained using six primers: Intseq1 (5′-ATT GGA GGA AAT GAA CAA GT-3′; 4173–4192), Intseq2 (5′-AGC AGA AGT TAT TCC AGC AG-3′; 4484–4503), INT-3 (5′-TTC GGG TTT ATT ACA G-3′; 4897–4912), INT-4 (5′-CTT GTA TTA CTA CTG C-3′; 4986–4971), Intseq-5 (5′-CTG GCT ACA TGA ACT GCT AC-3′; 4470–4452) and 3′INinR2 (5′-GCT TTC ATA GTG ATG TCT ATA AAA CC-3′; 5178–5153). Sequence editing and contig assembly were performed using SeqScape v2.6 (Applied Biosystems) with HXB2 as a reference sequence.
Next-generation sequencing and data analysis
To examine the mutation frequencies within the viral integrase by pyrosequencing, RNA was reverse transcribed and amplified in a single step in a touch-down PCR using bar-coded primers to enable 454 pyrosequencing with pooled amplicons. The integrase core domain from amino acid position 53 to amino acid 180 was analyzed. All mutations in the viral integrase at 18 different codons associated with raltegravir resistance [16, 20, 23, 45, 49–52] were evaluated and included amino acids: T66, L68, V72, L74, E92, Q95, T97, F121, E138, G140, Y143, Q148, V151, N155, K156, E157, K160 and G163.
All amplicons were purified with AMPure magnetic beads (Agencourt, Beckman Coulter, Krefeld, Germany), quality checked and quantified using an Agilent 2100 Bioanalyser (Agilent Life Sciences, Waldbronn, Germany) and picogreen using the fluorometer Fluostar Optima (BMG Labtech, Offenburg, Germany), respectively. After equimolar pooling of the amplicons, emulsion PCRs were performed. Pyrosequencing was done using primers A and B (Titanium emPCR kit Lib-L v2; Roche-454 Life Sciences, Branford, CT, USA). After bead recovery and enrichment, approximately 250,000 beads per pool were loaded on one region of a GS FLX PicoTiter plate subdivided with a four-lane gasket. Pyrosequencing was performed on a Genome Sequencer FLX (Roche-454 Life Sciences). Sequence readings from the 454 pyrosequencing run were extracted directly from the Standard-Flowgram-Files (sff). Reads were pair-wise aligned against the integrase sequence of reference strain HXB2. Multiple mutations present in a single read were assumed to originate from the same genome.
The dynamics of the sequence populations were investigated on de-aligned sets to avoid artifacts due to inconsistent alignment gap placements using statistical functions in R . For the analyses of the dominant variants in the populations, the redundancy-level for calling a variant was set at 80, i.e. only variants detected at least 80 times were considered. This level resulted in a strict filter that removed all known 454-sequencing artifacts [54–56]. The resulting dominant variants were aligned using MAFFT  and distance matrices were estimated using the R library ape  under a F84 substitution model (the choice of substitution model had no effect on the subsequent analyses). The R library sna  was used to construct minimum spanning trees (MSTs) based on the distance matrices of patient’s HIV population. Nodes were scaled according to both viral load and the relative abundance of each detected variant at time of sampling. We tracked known resistance mutations on the edges and edge lengths were drawn arbitrary in order to make the resulting graphs easy to look at.
Construction of deletion clone HXB2∆INT
An HXB2 molecular clone (pHXB2AF) was used to construct a molecular deletion clone lacking the integrase coding region. pHXB2AF is derived from pHXB2WT , which expresses the full length HIV-1 sequence HXB2 (9719 bp, Genbank accession number K03455.1), with all bacterial sequences non-essential for bacterial expression and replication removed.
The NdeI site present in Gp120 (at HXB2 nt 6404) was inactivated to create a unique NdeI site at the 3′ end of the integrase CDS. Therefore pHXB2AF was digested with NcoI (Roche Diagnostics, Almere, The Netherlands) and NheI (New England Biolabs, Ipswich, MA, USA) to remove a 1586 bp fragment containing the NdeI site. PCR, using VentR ® DNA polymerase (New England BioLabs) was performed on pHXB2AF with primers, NcoI-out (5′ CAC TAG AGC TTT TAG AGG AGC TTA AGA-3′; 5614–5640), NheI-out (5′-TTT TAT TAT TTC CAA ATT GTT CTC TTA-3′; 7296–7270) and NdeI-KO (5′-TCA GAT GCT AAA GCG TAT GAT ACA G-3′; 6390–6414). Primer NdeI-KO contained one silent nucleotide change (underlined) to inactivate the NdeI site in the PCR fragment. The amount of amplified product was further increased and enriched by performing a second amplification using VentR ® DNA polymerase with primers NcoI-in (5′-GAG CTT TTA GAG GAG CTT AAG AAT GAA-3′; 5619–5645), NheI-in (5′-ATT GTT CTC TTA ATT TGC TAG CTA TCT-3′; 7281–7255) and NdeI-KO. This PCR fragment was then digested with NcoI and NheI and ligated with the digested pHXB2AF, resulting in pHXB2AFNdeIKO which was confirmed by sequence analysis of the complete fragment.
Subsequently, the integrase coding region was removed from pHXB2AFNdeIKO. Therefore pHXB2AFNdeIKO was digested with MluNI (Roche) and NdeI (New England BioLabs). The fragment between MluNI and the 5′ end of integrase was restored by performing a PCR on pHXB2AF using primers RT19 (5′-GGA CAT AAA GCT ATA GGT ACA G-3′; 2454–2472) and NgoMIV-INTlinker (5′-TAA TAT CAT ATG GAC AGC GTC GCC GGC ACT GAC TAA TTT ATC TAC TTG TTC-3′). Primer NgoMIV-INTlinker contained two silent nucleotide changes with respect to pHXB2AFNdeIKO, thereby introducing a unique NgoMIV site (underlined 4209–4214) in the PCR fragment. In addition to these nucleotide changes the primer NgoMIV-INTlinker contains a linker sequence with a unique NdeI site and an AspI site. AspI was used to prevent re-ligation of the vector and the linker. The amount of amplified product was further increased and enriched by performing a second amplification using VentR ® DNA polymerase with primers RT19new2 (5′-GGA CCT ACA CCT GTC AAC ATA ATT GG-3′; 2484–2509) and NgoMIV-INTlinker. The PCR fragment was then digested with MluNI and NdeI and ligated with the digested pHXB2AFNdeIKO, resulting in a molecular deletion clone lacking the integrase coding region (pHXB2AFΔINT), which was confirmed by sequencing the complete fragment.
Generation of recombinant virus
To generate recombinant viruses, the second PCR as described in the population sequencing section, was performed with a different primer pair: forward primer NgoMIV-Int2 (5′-TTA GTC AGT GCC GGC ATC AGG AAA G-3′; 4200–4224) which contains a NgoMIV restriction site (underlined) and reverse primer 3INinR2. The obtained integrase fragment and vector pHXB2AFΔINT were digested with restriction enzymes NgoMIV (New England BioLabs) and NdeI. The PCR product and vector pHXB2AFΔINT were ligated overnight at 4 °C using the Rapid Ligation System (Promega, Benelux, Leiden, The Netherlands). After ligation, plasmids were digested with AspI and subsequently transformed into competent cells. Bacteria were cultured overnight at 37 °C. Plasmid isolation was performed using the QIAgen Plasmid Mini kit (Qiagen).
Viruses were generated by transfecting 293T cells with 10 ug plasmid DNA using Lipofectamine 2000 reagent (Invitrogen) according the manufacturer’s protocol. Two days after infection cell free virus was harvested and the viral infectivity (TCID50) was determined using end-point dilutions in MT2 cells.
Viral and cell culture
SupT1 and MT-2 cells were maintained in RPMI 1640 medium with l-glutamine (Lonza, Verviers, Belgium) supplemented with 10% FBS (FBS; Sigma-Aldrich, Zwijndrecht, The Netherlands) and 10 µg/ml gentamicin (Invitrogen, Breda, The Netherlands). 293T cells were maintained in DMEM with l-glutamine (Lonza) supplemented with 10% FBS and 10 µg/ml gentamicin.
In vitro selection experiments
The raltegravir in vitro selection experiments with two recombinant viruses that contained patient-derived integrase CDS and HIV-1 reference strain HXB2 were each performed 5 times. The in vitro selections were initiated by infecting 2 × 106 SupT1 cells at a multiplicity of infection (MOI) of 0.001. The initial raltegravir concentration was 2 nM Raltegravir. Cultures were monitored daily for cytopatic effect (CPE) and twice a week half of the medium was replaced by fresh medium supplemented with raltegravir. When full blown CPE was observed, cell free virus was harvested. The raltegravir concentration was raised in each passage to a final concentration of 1024 nM in passage 10. After passage 10, HIV-1 RNA was isolated from all cultures for genotypic analysis. As a control, in vitro evolution experiments (10 passages) were performed with HXB2 reference strain (5 individual cultures) to monitor the evolution of the integrase CDS during culture in absence of inhibitor.
Phenotypic drug susceptibility analysis
Drug susceptibility was determined by a multiple cycle cell-killing assay . MT-2 cells (4 × 104 in 200 µl RPMI 10% FBS per well) were plated in 96-well microplates. Sample virus and reference virus were inoculated for five days on a single 96-well plate in the presence of threefold dilutions of raltegravir. Both sample virus and reference virus were inoculated at multiple MOIs to adjust for any differences in viral RC. Fold change (FC) values were calculated by dividing the mean 50% inhibitory concentration (EC50) for a sample virus by that of the HXB2 reference strain.
AF, LV, AW and MN designed the study. MD, BB, IH and AW collected patient information and provided samples or sequences. AF, MD, AT and PS generated the data. AF, TL, AW and MN analyzed and interpreted the data and wrote the manuscript. All authors read and approved the final manuscript.
Raltegravir was obtained through the NIH AIDS Reagent Program, Division of AIDS, NIAID, NIH: Raltegravir (Cat # 11680) from Merck & Company, Inc.
The authors declare no competing financial or other interests.
Availability of data and material
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Consent for publication
Ethics approval and consent to participate
Patients 1, 2 and 5 are from the Netherlands and are part of the observational AIDS Therapy Evaluation in the Netherlands (ATHENA) cohort, which follows HIV-positive patients who are registered in on the designated treatment centers in the Netherlands. Patients can opt-out after being informed on the purpose of data collection by their treating physician. Patients who have decided not to opt out are anonymously recorded in a central database. Patients 3 and 4 are from Germany and their genotypic resistance tests were done in the scope of routine diagnostics and no extra blood draws were performed.
This work was supported by (AF) the Dutch AIDS Fund (Project Number 2006028), (MN) the Netherlands Organization for Scientific Research (NOW VIDI Grant 91796349) and TL was supported by the National Institutes of Health (NIH Grant R01AI087520). The funders had no role in study design, collecting, analyzing or interpreting the data, or in the preparation of the manuscript.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Palella FJ Jr, Delaney KM, Moorman AC, Loveless MO, Fuhrer J, Satten GA, Aschman DJ, Holmberg SD. Declining morbidity and mortality among patients with advanced human immunodeficiency virus infection. HIV Outpatient study investigators. N Engl J Med. 1998;338:853–60.View ArticlePubMedGoogle Scholar
- Boyer JC, Bebenek K, Kunkel TA. Unequal human immunodeficiency virus type 1 reverse transcriptase error rates with RNA and DNA templates. Proc Natl Acad Sci USA. 1992;89:6919–23.View ArticlePubMedPubMed CentralGoogle Scholar
- Ji JP, Loeb LA. Fidelity of HIV-1 reverse transcriptase copying RNA in vitro. Biochemistry. 1992;31:954–8.View ArticlePubMedGoogle Scholar
- Menendez-Arias L. Molecular basis of fidelity of DNA synthesis and nucleotide specificity of retroviral reverse transcriptases. Prog Nucleic Acid Res Mol Biol. 2002;71:91–147.View ArticlePubMedGoogle Scholar
- Bonhoeffer S, May RM, Shaw GM, Nowak MA. Virus dynamics and drug therapy. Proc Natl Acad Sci USA. 1997;94:6971–6.View ArticlePubMedPubMed CentralGoogle Scholar
- Nowak MA, Bonhoeffer S, Shaw GM, May RM. Anti-viral drug treatment: dynamics of resistance in free virus and infected cell populations. J Theor Biol. 1997;184:203–17. https://doi.org/10.1006/jtbi.1996.0307.View ArticlePubMedGoogle Scholar
- Nijhuis M, Boucher CA, Schipper P, Leitner T, Schuurman R, Albert J. Stochastic processes strongly influence HIV-1 evolution during suboptimal protease-inhibitor therapy. Proc Natl Acad Sci USA. 1998;95:14441–6.View ArticlePubMedPubMed CentralGoogle Scholar
- Duffy S, Shackelton LA, Holmes EC. Rates of evolutionary change in viruses: patterns and determinants. Nat Rev Genet. 2008;9:267–76. https://doi.org/10.1038/nrg2323.View ArticlePubMedGoogle Scholar
- Alexander HK, Bonhoeffer S. Pre-existence and emergence of drug resistance in a generalized model of intra-host viral dynamics. Epidemics. 2012;4:187–202. https://doi.org/10.1016/j.epidem.2012.10.001.View ArticlePubMedGoogle Scholar
- Espeseth AS, Felock P, Wolfe A, Witmer M, Grobler J, Anthony N, Egbertson M, Melamed JY, Young S, Hamill T, et al. HIV-1 integrase inhibitors that compete with the target DNA substrate define a unique strand transfer conformation for integrase. Proc Natl Acad Sci USA. 2000;97:11244–9. https://doi.org/10.1073/pnas.200139397.View ArticlePubMedPubMed CentralGoogle Scholar
- Hare S, Gupta SS, Valkov E, Engelman A, Cherepanov P. Retroviral intasome assembly and inhibition of DNA strand transfer. Nature. 2010;464:232–6. https://doi.org/10.1038/nature08784.View ArticlePubMedPubMed CentralGoogle Scholar
- Krishnan L, Li X, Naraharisetty HL, Hare S, Cherepanov P, Engelman A. Structure-based modeling of the functional HIV-1 intasome and its inhibition. Proc Natl Acad Sci USA. 2010;107:15910–5. https://doi.org/10.1073/pnas.1002346107.View ArticlePubMedPubMed CentralGoogle Scholar
- ISENTRESS™ (raltegravir) for treatment of HIV (NDA 22145 S036). FDA May 31 2017. https://www.accessdata.fda.gov/drugsatfda_docs/label/2017/022145s036,203045s013,205786s004lbl.pdf.
- Messiaen P, Wensing AM, Fun A, Nijhuis M, Brusselaers N, Vandekerckhove L. Clinical use of HIV integrase inhibitors: a systematic review and meta-analysis. PLoS ONE. 2013;8:e52562. https://doi.org/10.1371/journal.pone.0052562.View ArticlePubMedPubMed CentralGoogle Scholar
- Rockstroh JK, DeJesus E, Lennox JL, Yazdanpanah Y, Saag MS, Wan H, Rodgers AJ, Walker ML, Miller M, DiNubile MJ, et al. Durable efficacy and safety of raltegravir versus efavirenz when combined with tenofovir/emtricitabine in treatment-naive HIV-1-infected patients: final 5-year results from STARTMRK. J Acquir Immune Defic Syndr. 2013;63:77–85. https://doi.org/10.1097/QAI.0b013e31828ace69.View ArticlePubMedGoogle Scholar
- Cooper DA, Steigbigel RT, Gatell JM, Rockstroh JK, Katlama C, Yeni P, Lazzarin A, Clotet B, Kumar PN, Eron JE, et al. Subgroup and resistance analyses of raltegravir for resistant HIV-1 infection. N Engl J Med. 2008;359:355–65.View ArticlePubMedGoogle Scholar
- Miller MD, Danovich R, Ke Y, Witmer M, Zhao J, Harvey C, Nguyen BY, Hazuda DJ: Longitudinal analysis of resistance to the HIV-1 integrase inhibitor raltegravir: results from P005 a phase 2 study in treatment experienced patients. In: XVII international HIV drug resistance workshop, Sitges, Spain. 2008.Google Scholar
- Johnson VA, Brun-Vezinet F, Clotet B, Gunthard HF, Kuritzkes DR, Pillay D, Schapiro JM, Richman DD. Update of the drug resistance mutations in HIV-1: December 2010. Top HIV Med. 2010;18:156–63.PubMedGoogle Scholar
- Fransen S, Gupta S, Danovich R, Hazuda D, Miller M, Witmer M, Petropoulos CJ, Huang W. Loss of raltegravir susceptibility by human immunodeficiency virus type 1 is conferred via multiple nonoverlapping genetic pathways. J Virol. 2009;83:11440–6.View ArticlePubMedPubMed CentralGoogle Scholar
- Malet I, Delelis O, Valantin MA, Montes B, Soulie C, Wirden M, Tchertanov L, Peytavin G, Reynes J, Mouscadet JF, et al. Mutations associated with failure of raltegravir treatment affect integrase sensitivity to the inhibitor in vitro. Antimicrob Agents Chemother. 2008;52:1351–8.View ArticlePubMedPubMed CentralGoogle Scholar
- Nakahara K, Wakasa-Morimoto C, Kobayashi M, Miki S, Noshi T, Seki T, Kanamori-Koyama M, Kawauchi S, Suyama A, Fujishita T, et al. Secondary mutations in viruses resistant to HIV-1 integrase inhibitors that restore viral infectivity and replication kinetics. Antivir Res. 2009;81:141–6.View ArticlePubMedGoogle Scholar
- Quercia R, Dam E, Perez-Bercoff D, Clavel F. Selective-advantage profile of human immunodeficiency virus type 1 integrase mutants explains in vivo evolution of raltegravir resistance genotypes. J Virol. 2009;83:10245–9.View ArticlePubMedPubMed CentralGoogle Scholar
- Fun A, Van Baelen K, van Lelyveld SF, Schipper PJ, Stuyver LJ, Wensing AM, Nijhuis M. Mutation Q95K enhances N155H-mediated integrase inhibitor resistance and improves viral replication capacity. J Antimicrob Chemother. 2010;65:2300–4.View ArticlePubMedGoogle Scholar
- Blanco JL, Varghese V, Rhee SY, Gatell JM, Shafer RW. HIV-1 integrase inhibitor resistance and its clinical implications. J Infect Dis. 2011;203:1204–14.View ArticlePubMedPubMed CentralGoogle Scholar
- Anstett K, Brenner B, Mesplede T, Wainberg MA. HIV drug resistance against strand transfer integrase inhibitors. Retrovirology. 2017;14:36. https://doi.org/10.1186/s12977-017-0360-7.View ArticlePubMedPubMed CentralGoogle Scholar
- Malet I, Delelis O, Soulie C, Wirden M, Tchertanov L, Mottaz P, Peytavin G, Katlama C, Mouscadet JF, Calvez V, Marcelin AG. Quasispecies variant dynamics during emergence of resistance to raltegravir in HIV-1-infected patients. J Antimicrob Chemother. 2009;63:795–804.View ArticlePubMedGoogle Scholar
- Reigadas S, Anies G, Masquelier B, Calmels C, Stuyver LJ, Parissi V, Fleury H, Andreola ML. The HIV-1 integrase mutations Y143C/R are an alternative pathway for resistance to Raltegravir and impact the enzyme functions. PLoS ONE. 2010;5:e10311. https://doi.org/10.1371/journal.pone.0010311.View ArticlePubMedPubMed CentralGoogle Scholar
- da Silva D, Van Wesenbeeck L, Breilh D, Reigadas S, Anies G, Van Baelen K, Morlat P, Neau D, Dupon M, Wittkop L, et al. HIV-1 resistance patterns to integrase inhibitors in antiretroviral-experienced patients with virological failure on raltegravir-containing regimens. J Antimicrob Chemother. 2010;65:1262–9. https://doi.org/10.1093/jac/dkq099.View ArticlePubMedGoogle Scholar
- Mukherjee R, Jensen ST, Male F, Bittinger K, Hodinka RL, Miller MD, Bushman FD. Switching between raltegravir resistance pathways analyzed by deep sequencing. AIDS. 2011;25:1951–9. https://doi.org/10.1097/QAD.0b013e32834b34de.View ArticlePubMedPubMed CentralGoogle Scholar
- Goethals O, Vos A, Van Ginderen M, Geluykens P, Smits V, Schols D, Hertogs K, Clayton R. Primary mutations selected in vitro with raltegravir confer large fold changes in susceptibility to first-generation integrase inhibitors, but minor fold changes to inhibitors with second-generation resistance profiles. Virology. 2010;402:338–46.View ArticlePubMedGoogle Scholar
- Metifiot M, Vandegraaff N, Maddali K, Naumova A, Zhang X, Rhodes D, Marchand C, Pommier Y. Elvitegravir overcomes resistance to raltegravir induced by integrase mutation Y143. Aids. 2011;25:1175–8.View ArticlePubMedGoogle Scholar
- Castagna A, Maggiolo F, Penco G, Wright D, Mills A, Grossberg R, Molina JM, Chas J, Durant J, Moreno S, et al. Dolutegravir in antiretroviral-experienced patients with raltegravir- and/or elvitegravir-resistant HIV-1: 24-week results of the phase III VIKING-3 study. J Infect Dis. 2014;210:354–62. https://doi.org/10.1093/infdis/jiu051.View ArticlePubMedPubMed CentralGoogle Scholar
- Akil B, Blick G, Hagins DP, Ramgopal MN, Richmond GJ, Samuel RM, Givens N, Vavro C, Song IH, Wynne B, Ait-Khaled M. Dolutegravir versus placebo in subjects harbouring HIV-1 with integrase inhibitor resistance associated substitutions: 48-week results from VIKING-4, a randomized study. Antivir Ther. 2015;20:343–8. https://doi.org/10.3851/IMP2878.View ArticlePubMedGoogle Scholar
- Hofstra LM, Nijhuis M, Mudrikova T, Fun A, Schipper P, Schneider M, Wensing A. Use of dolutegravir in two INI-experienced patients with multiclass resistance resulted in excellent virological and immunological responses. J Int AIDS Soc. 2014;17:19755. https://doi.org/10.7448/IAS.17.4.19755.View ArticlePubMedGoogle Scholar
- Seki T, Kobayashi M, Wakasa-Morimoto C, Yoshinaga T, Sato A, Fujiwara T, Underwood MR, Garvey EP, Johns BA: S/GSK1349572 is a potent next generation HIV integrase inhibitor and demonstrates a superior resistance profile substantiated with 60 integrase mutant molecular clones. In: 17th conference on retroviruses and opportunistic infections, San Francisco, CA, USA. 2010.Google Scholar
- Brenner BG, Wainberg MA. Clinical benefit of dolutegravir in HIV-1 management related to the high genetic barrier to drug resistance. Virus Res. 2017;239:1–9. https://doi.org/10.1016/j.virusres.2016.07.006.View ArticlePubMedGoogle Scholar
- Laskey SB, Siliciano RF. Quantitative evaluation of the antiretroviral efficacy of dolutegravir. JCI Insight. 2016;1:e90033. https://doi.org/10.1172/jci.insight.90033.View ArticlePubMedPubMed CentralGoogle Scholar
- Malet I, Gimferrer Arriaga L, Artese A, Costa G, Parrotta L, Alcaro S, Delelis O, Tmeizeh A, Katlama C, Valantin MA, et al. New raltegravir resistance pathways induce broad cross-resistance to all currently used integrase inhibitors. J Antimicrob Chemother. 2014;69:2118–22. https://doi.org/10.1093/jac/dku095.View ArticlePubMedGoogle Scholar
- Brenner BG, Thomas R, Blanco JL, Ibanescu RI, Oliveira M, Mesplede T, Golubkov O, Roger M, Garcia F, Martinez E, Wainberg MA. Development of a G118R mutation in HIV-1 integrase following a switch to dolutegravir monotherapy leading to cross-resistance to integrase inhibitors. J Antimicrob Chemother. 2016;71:1948–53. https://doi.org/10.1093/jac/dkw071.View ArticlePubMedPubMed CentralGoogle Scholar
- Liu J, Miller MD, Danovich RM, Vandergrift N, Cai F, Hicks CB, Hazuda DJ, Gao F. Analysis of low-frequency mutations associated with drug resistance to raltegravir before antiretroviral treatment. Antimicrob Agents Chemother. 2010;55:1114–9. https://doi.org/10.1128/AAC.01492-10.View ArticlePubMedPubMed CentralGoogle Scholar
- Charpentier C, Laureillard D, Piketty C, Tisserand P, Batisse D, Karmochkine M, Si-Mohamed A, Weiss L. High frequency of integrase Q148R minority variants in HIV-infected patients naive of integrase inhibitors. AIDS. 2010;24:867–73. https://doi.org/10.1097/QAD.0b013e3283367796.View ArticlePubMedGoogle Scholar
- Codoner FM, Pou C, Thielen A, Garcia F, Delgado R, Dalmau D, Santos JR, Buzon MJ, Martinez-Picado J, Alvarez-Tejado M, et al. Dynamic escape of pre-existing raltegravir-resistant HIV-1 from raltegravir selection pressure. Antivir Res. 2010;88:281–6. https://doi.org/10.1016/j.antiviral.2010.09.016.View ArticlePubMedGoogle Scholar
- Johnson JA, Geretti AM. Low-frequency HIV-1 drug resistance mutations can be clinically significant but must be interpreted with caution. J Antimicrob Chemother. 2010;65:1322–6. https://doi.org/10.1093/jac/dkq139.View ArticlePubMedGoogle Scholar
- Li JZ, Paredes R, Ribaudo HJ, Svarovskaia ES, Metzner KJ, Kozal MJ, Hullsiek KH, Balduin M, Jakobsen MR, Geretti AM, et al. Low-frequency HIV-1 drug resistance mutations and risk of NNRTI-based antiretroviral treatment failure: a systematic review and pooled analysis. JAMA. 2011;305:1327–35. https://doi.org/10.1001/jama.2011.375.View ArticlePubMedPubMed CentralGoogle Scholar
- Kobayashi M, Nakahara K, Seki T, Miki S, Kawauchi S, Suyama A, Wakasa-Morimoto C, Kodama M, Endoh T, Oosugi E, et al. Selection of diverse and clinically relevant integrase inhibitor-resistant human immunodeficiency virus type 1 mutants. Antivir Res. 2008;80:213–22.View ArticlePubMedGoogle Scholar
- Hu Z, Kuritzkes DR. Effect of raltegravir resistance mutations in HIV-1 integrase on viral fitness. J Acquir Immune Defic Syndr. 2010;55:148–55. https://doi.org/10.1097/QAI.0b013e3181e9a87a.View ArticlePubMedPubMed CentralGoogle Scholar
- Weir JT, Haddrath O, Robertson HA, Colbourne RM, Baker AJ. Explosive ice age diversification of kiwi. Proc Natl Acad Sci USA. 2016;113:E5580–7. https://doi.org/10.1073/pnas.1603795113.View ArticlePubMedPubMed CentralGoogle Scholar
- Halliday TJD, Goswami A. Eutherian morphological disparity across the end-Cretaceous mass extinction. Biol J Lin Soc. 2016;118:152–68.View ArticleGoogle Scholar
- Charpentier C, Karmochkine M, Laureillard D, Tisserand P, Belec L, Weiss L, Si-Mohamed A, Piketty C. Drug resistance profiles for the HIV integrase gene in patients failing raltegravir salvage therapy. HIV Med. 2008;9:765–70.View ArticlePubMedGoogle Scholar
- Goodman D, Hluhanich R, Waters J, Margot NA, Fransen S, Gupta S, Huang W, Parkin N, Borroto-Esoda K, Svarovskaia ES, et al: Integrase inhibitor resistance involves complex interactions among primary and secondary resistance mutations: a novel mutation L68V/I associates with E92Q and increases resistance. In: XVII international HIV drug resistance workshop, Sitges, Spain. 2008.Google Scholar
- Markowitz M, Nguyen BY, Gotuzzo E, Mendo F, Ratanasuwan W, Kovacs C, Prada G, Morales-Ramirez JO, Crumpacker CS, Isaacs RD, et al. Rapid and durable antiretroviral effect of the HIV-1 integrase inhibitor raltegravir as part of combination therapy in treatment-naive patients with HIV-1 infection: results of a 48-week controlled study. J Acquir Immune Defic Syndr. 2007;46:125–33.View ArticlePubMedGoogle Scholar
- Sichtig N, Sierra S, Kaiser R, Daumer M, Reuter S, Schulter E, Altmann A, Fatkenheuer G, Dittmer U, Pfister H, Esser S. Evolution of raltegravir resistance during therapy. J Antimicrob Chemother. 2009;64:25–32.View ArticlePubMedGoogle Scholar
- Team RDC: R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org. 2008.
- Brodin J, Mild M, Hedskog C, Sherwood E, Leitner T, Andersson B, Albert J. PCR-induced transitions are the major source of error in cleaned ultra-deep pyrosequencing data. PLoS ONE. 2013;8:e70388. https://doi.org/10.1371/journal.pone.0070388.View ArticlePubMedPubMed CentralGoogle Scholar
- Hedskog C, Mild M, Jernberg J, Sherwood E, Bratt G, Leitner T, Lundeberg J, Andersson B, Albert J. Dynamics of HIV-1 quasispecies during antiviral treatment dissected using ultra-deep pyrosequencing. PLoS ONE. 2010;5:e11345. https://doi.org/10.1371/journal.pone.0011345.View ArticlePubMedPubMed CentralGoogle Scholar
- Tsibris AM, Korber B, Arnaout R, Russ C, Lo CC, Leitner T, Gaschen B, Theiler J, Paredes R, Su Z, et al. Quantitative deep sequencing reveals dynamic HIV-1 escape and large population shifts during CCR5 antagonist therapy in vivo. PLoS ONE. 2009;4:e5683. https://doi.org/10.1371/journal.pone.0005683.View ArticlePubMedPubMed CentralGoogle Scholar
- Katoh K, Asimenos G, Toh H. Multiple alignment of DNA sequences with MAFFT. Methods Mol Biol. 2009;537:39–64. https://doi.org/10.1007/978-1-59745-251-9_3.View ArticlePubMedGoogle Scholar
- Paradis E, Claude J, Strimmer K. APE: analyses of phylogenetics and evolution in R language. Bioinformatics. 2004;20:289–90.View ArticlePubMedGoogle Scholar
- Butts SF, Owen C, Mainigi M, Senapati S, Seifer DB, Dokras A. Assisted hatching and intracytoplasmic sperm injection are not associated with improved outcomes in assisted reproduction cycles for diminished ovarian reserve: an analysis of cycles in the United States from 2004 to 2011. Fertil Steril. 2014;102(1041–1047):e1041. https://doi.org/10.1016/j.fertnstert.2014.06.043.View ArticleGoogle Scholar
- van Maarseveen NM, Huigen MC, de Jong D, Smits AM, Boucher CA, Nijhuis M. A novel real-time PCR assay to determine relative replication capacity for HIV-1 protease variants and/or reverse transcriptase variants. J Virol Methods. 2006;133:185–94. https://doi.org/10.1016/j.jviromet.2005.11.008.View ArticlePubMedGoogle Scholar
- Boucher CA, Keulen W, van Bommel T, Nijhuis M, de Jong D, de Jong MD, Schipper P, Back NK. Human immunodeficiency virus type 1 drug susceptibility determination by using recombinant viruses generated from patient sera tested in a cell-killing assay. Antimicrob Agents Chemother. 1996;40:2404–9.PubMedPubMed CentralGoogle Scholar