Colorectal cancers express EGF and/or EGFR mRNA in 66% of primary tumors, 44% of adjacent mucosa, and up to 62% of positive lymph nodes. Patients enrolled in NSABP FR-1 will begin therapy with the anti-EGFR antibody panitumumab--which has shown single-agent activity in colorectal cancer patients--in combination with CAPOX, an effective treatment for patients with advanced colorectal cancer, with the convenience of using an oral fluoropyrimidine (capecitabine.) The intent with this first phase is to expose distant metastases to the agents early and to shrink the tumor before giving radiation therapy. After chemotherapy, radiotherapy with capecitabine and panitumumab will be given to destroy the primary tumor. Panitumumab is added to the radiochemotherapy regimen because data showed positive results when another anti-EGFR antibody was added to radiotherapy in patients with head and neck cancer. This approach aims to improve neoadjuvant combined modality therapy for rectal cancer; it should provide effective therapy for eligible patients and valuable information about clinical tumor response rates and treatment tolerability, which, if favorable, may be used to develop future Phase III trials.