Notch - the protein that can help determine cell fate - maintains a stable population of basal cells in the prostate through a positive feedback loop system with another key protein - TGF beta (transforming growth factor beta), said Baylor College of Medicine researchers in the journal Cell Stem Cell.

"When basal cell homeostasis (or maintenance of a stable population) is disrupted, it may be part of the process that initiates prostate cancer," said Dr. Li Xin, assistant professor of molecular and cellular biology at BCM and a senior author of the report.

"Notch signaling takes different functions in different cell lineages," he said. "For example, Notch can suppress proliferation of basal cells and enhance it in luminal cells (another population of cells in the prostate)."

In studies in mice, Xin and his colleagues found that disrupting the Notch signal did not have much effect on the basal cells. However, when they disrupted both Notch and TGF beta signal, the basal cells begin to proliferate a lot," he said.

Notch is one of the important parts of the process by which basal cells sense the signals from TGF beta, said Xin.

While it might be tempting to suppress Notch signaling in an effort to prevent prostate cancer, it is probably not the best course. Systemic long-term inhibition of Notch can be toxic to intestines and result in vascular tumors.

"Once the important layer of basal cells is laid down in the prostate, the Notch-TGF beta feedback loop keeps the homeostasis of basal cells in the adult mouse prostate," said Xin.

"The disruption of this layer of basal cells may play a critical role in the initiation of prostate cancer," said Xin.