A Phase II Study Evaluating an Organ Preservation Strategy Using Immune Checkpoint Blockade for Participants With Primary Colorectal or Gastroesophageal Cancer

Background: * Immune checkpoint blockade (ICB) using antibodies against programmed cell death protein 1 (PD-1) and cytotoxic T lymphocyte associated protein 4 (CTLA-4) can induce major pathologic responses (MPR) in approximately 23% of mismatch repair (MMR) proficient colorectal cancers, 90% of MMR deficient colorectal cancers and 74% of MMR deficient gastroesophageal cancers. * MPR after ICB, defined as \>90% treatment response, is associated with exceptional local and distant disease-free survival (DFS). This has been observed in participants with advanced melanoma, MMR deficient colorectal cancer and MMR deficient gastroesophageal cancer. * We hypothesize that it is safe to forego surgical resection in participants who have had an MPR to ICB. To test this hypothesis, we designed an organ preservation strategy for participants with colorectal cancer and gastroesophageal cancer using induction ICB and close interval surveillance. Objectives: Primary objective: --Determine the rate of clinical complete response (CR) or near-complete response (nCR) after induction ICB in participants with MMR proficient colorectal cancer (Cohort 1), MMR deficient colorectal cancer (Cohort 2) and MMR deficient gastroesophageal cancer (Cohort 3). Eligibility: * Age \>= 18 years * Biopsy-confirmed stage I-III colorectal cancer (MMR proficient or MMR deficient) or stage I-III gastroesophageal cancer (MMR deficient only) * ECOG 0-1 * May not have allergies or hypersensitivities to anti-PD-1 or anti-CTLA-4 administration * No concurrent major medical illnesses * No history of grade III or IV irAEs affecting major organ systems associated with the administration of single agent anti-PD-1, anti-PD-L1 or anti-CTLA-4 antibodies * Adequate organ function Design: * This is a phase II, single center study evaluating induction PD-1 blockade with or without CTLA-4 blockade in participants with primary colorectal or gastroesophageal cancer. * Participants will undergo baseline complete endoscopy with biopsies, scans and labs. Apheresis will also be performed before treatment initiation. * All participants will receive nivolumab (3 mg/kg) every 3 weeks for an initial 4 cycles and may be eligible for an additional 4 cycles depending on response, for a total of 8 cycles. * Participants in Cohort 1 (MMR proficient colorectal cancers) and Cohort 3 (MMR deficient gastroesophageal cancers) will receive low-dose ipilimumab at 1 mg/kg every 6 weeks for 2 cycles, and may be eligible for an additional 2 cycles depending on response, for a maximum of 4 cycles. * Participants will be dosed with nivolumab +/- ipilimumab every three weeks at the NIH Clinical Center. A safety evaluation will be performed before each dose including history, physical exam and laboratory tests. * Radiographic and endoscopic evaluation for response will be every 6 weeks while on study (6, 12, 18 and 24 weeks). The assessment at each timepoint will dictate further management (observation, continuation of therapy or surgery/ standard chemo/ chemoradiation). * After a maximum of 8 cycles (24 weeks): * Participants with CR or nCR will be followed on a standardized surveillance protocol consisting of physical examination, cross-sectional imaging and flexible endoscopy. * All other participants will be recommended to undergo surgical resection or will be referred for other standard treatments if available (i.e. chemotherapy or radiotherapy if indicated).