Common problems in all studies

We agree with the authors of the meta-analysis3 that the lack of a placebo-controlled and blinded study affects the validity of the three studies and consequently the validity of the review. Without placebo control, it is impossible to differentiate between specific pharmacological and placebo effects. Placebo effect is defined as the ‘response of a subject to a substance or any procedure known to be without specific therapeutic effect for the condition being treated’.23 Several studies demonstrated that perceptual characteristics of drugs,24 the route of administration,25 laboratory tests,26 diagnosis,27 and doctor–patient relationship play an important role in the outcome of illness.28–31 Information regarding treatment or no treatment alone is sufficient to cause a placebo effect.32 Moreover, patients and doctors’ preferences could also have influenced the results in an open study.33 Patients assigned to the control group feel disadvantaged because they expect to be treated. Furthermore, when there is no concealment of treatment allocation, the randomisation procedure is compromised because of conscious or subconscious bias.34 It is important to perform an ITT analysis to maintain the balance distribution of risk factors between groups which is achieved by a randomisation procedure. Only in the study of Sakuramoto et al 15 that a correct ITT analysis was conducted. Collectively, these aspects affect the validity of the reports and therefore the described effects cannot be considered as clinically relevant.

Specific problems of the study by Neri et al14

This study is the final report of a previously published study by Neri et al 35 ,35 in which patients were stratified by centre to receive either CTx or were in the control group at follow-up. It is not clear whether the centres were stratified and patients in each centre were randomised by a single study centre. Alternatively, each centre could have randomised its own patients. The latter procedure could also explain the unbalanced risk profiles (table 1 of the original article) in favour of the intervention group. Additionally, only evaluable patients were reported in the table and were analysed. Following these findings, a researcher bias is present in the study; in the title of the publication, the authors declared that the study was an RCT which could not be confirmed by our questionnaire however.

Specific problems of the study by Sakuramoto et al15

In the study of Sakuramoto et al,15 a minimisation method is used. Minimisation,36–39 a type of dynamic allocation, is gaining popularity especially in clinical cancer trials. In this design, the new subject’s treatment assignment is determined by evaluating the potential covariate imbalance that would result if he or she were assigned to the treatment or likewise to the control group.40 Minimisation aims at achieving balance over a large number of prespecified prognostic factors simultaneously. In contrast to the opinion of the authors of the Cochrane review,3 we raise concerns over this design as it compromises adequate generation of an allocation sequence and concealment in this study. In fact, investigators using minimisation can determine the group to which a prospective subject would be allocated and then decide whether this is positive or negative in terms of creating an imbalance in some key predictor of outcome not considered in the imbalance function. Despite adding randomisation, so that the treatment that minimises the imbalance function for a given patient is not necessarily allocated to that patient, there is a high probability of this being the case.41 The European Medicines Agency’s committee42 states that ‘dynamic allocation is strongly discouraged.’ Regarding follow-up modalities, the patients in the intervention group underwent more frequent haematological tests than patients in the control group. This could be a source of bias because any treatment and additional attention from the doctor (difference in care) could lead to an improvement in the patients’ outcome.43 Moreover, Sox et al 26 found that laboratory tests that have no diagnostic value were independent factors of recovery. Finally, a sponsor-related conflict of interest was identified by our analysis as also acknowledged in the Cochrane review.3

Specific problems of the study by Fujimoto et al16

In this study, similar to the study of Neri et al,14 the randomisation process is not described, and a table with the characteristics of the patients in the two groups is not reported. This makes it impossible to determine the balance which is a direct sign of a good randomisation process. Interestingly, only 75% of the patients included in the intervention group were analysed. The sample size is not sufficient to reach the needed power for the chosen significance levels provided.

Specific problems of the study by Douglass et al17

In this study, similar to the study of Neri et al,14 the randomisation process is not described. Twenty-three randomised patients were excluded by a committee and by the principal investigator from the final analysis. This results in a loss of power. Additionally, the reasons for the withdrawal of these patients are not explained well enough and conflict of interest cannot be excluded. A power calculation is not reported.

In 2010, the GASTRIC (Global Advanced/Adjuvant Stomach Tumor Research International) Group published a meta-analysis on the same topic as Diaz-Nieto et al 3 and similarly found a benefit of adjuvant CTx for resectable gastric cancer (HR 0.82, 95% CI (0.76 to 0.90)).44 The authors found a total of 31 eligible trials. After asking for individual patients’ data, they obtained data of 17 trials only. Thus, the performed individual patient-level meta-analysis included only 17 of the 31 studies. This meta-analysis has the advantage to be based on individual patient data and has the limitation to include only a part of the existing literature (17 of 31 studies, 55%). Consequently, the results should be treated with caution because they are only partial.

Generally, the discussion about quality and quality assessment of medical publications is still ongoing. When authoring a clinical study, it is important to describe the study according to the CONSORT checklist22 if the study is an RCT, or according to the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) checklist45 if it is observational. In the case of a meta-analysis, it is mandatory to check the validity of each publication and this check should be included in the review. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist46 helps authors to improve the quality of a meta-analysis of RCT while the MOOSE (Meta-analysis of Observational Studies in Epidemiology) guidelines47 instruct the process of meta-analysing observational studies. In several medical journals, the checklist that goes with the study type must be submitted together with the main manuscript. This increases the quality and standardisation of publications and it is recommended that this procedure becomes a standard of practice for each journal. Several journals also request the trial registration number to consider a study for publication and this is also recommended to become a standard of practice for all publication avenues.

For a reviewer, it is important to carefully evaluate a publication, in particular when a RCT is reported. Design, conduct, reporting, and statistics should be analysed in detail before acceptance. Only valid studies are reliable studies. For an expert pool aiming to publish guidelines, it is necessary to scrutinise the validity of single studies and of meta-analyses as well. As recently shown by Shnier et al,48 financial conflict of interest and relationships between guideline authors and drug companies are common and represent a source of bias in studies. As authoritative value is assigned to guidelines, it is important to develop formal policies to limit the potential influence of any conflict of interest on guideline recommendations.48 Only in this way it is possible to improve the quality of medical publications.

In the second part of our work, we performed the meta-analysis first without the four analysed studies and showed that the result of the meta-analysis does not differ from that of the original one. Moreover, when the other four studies with positive and statistically significant results were excluded as well and only 26 statistically non-significant studies were included, we still found a statistically significant meta-analysis estimate confirming the better survival in patients receiving postsurgical CTx in comparison to those undergoing only gastric resection. This is due to the fact that meta-analyses increase power as described in the Cochrane Handbook49: ‘many individual studies are too small to detect small effects, but when several are combined there is a higher chance of detecting an effect.’ This point is very critical as it means that a statistically significant result in a meta-analysis can be obtained even if none of the included studies found a statistically significant result. This finding again highlights the importance of including only studies of high quality in a meta-analysis, especially in the case of studies which did not find a statistically significant estimate. A meta-analysis can often find a statistically significant result just because of the increase of sample size independently from the quality of the included studies. Yet another example of a meta-analysis with a global statistic positive estimate even if all included studies are not statistically significant can be found in the literature50 and is already critically revised.51

Implications for practice

Following the results of the Cochrane review, postsurgical CTx should be used for patients with resectable gastric cancer. However, it is important to note that some of the included trials contain limitations so that definitive assessments of this topic should be delayed until future trials are properly developed. The four analysed studies that were chosen because of their attributed weights are not of sufficient validity to be included in a meta-analysis, which holds for most of the other studies included.

Perioperative CTx in patients with gastric cancer is now the standard treatment in most centres today and has found its recommendation in several national guidelines and has also been proposed by a Cochrane review.52 However, critical analysis of the leading publications that support perioperative CTx treatment53–55 uncovered serious shortcomings particularly with regard to patient selection, changes in protocol, homogeneity of subjects, surgical quality and analysis of the results.56 The authors of this critical appraisal conclude that none of these studies justify an unrestrained recommendation of perioperative CTx for advanced gastric cancer. Even more interesting, the same group found different recommendations in international guidelines on this topic, despite the fact that all guidelines claim to be based on the same publications.57

Although perioperative CTx in case of advanced gastric cancer is recommended, the literature needs to be reassessed in detail in order to avoid unnecessary side effects for the patients as well as unnecessary costs for the healthcare systems.