I found this article when I was doing some research for my James class. Richard Bauckham wrote a book that was required reading for the course (see my thoughts on it here), and he had some interesting things to say regarding the topic of the relationship between Jesus and his brothers (were they Mary’s kids, or just Joseph’s etc). I thought I would post part of it here just as a resource.
“The historical evidence is not sufficient for a firm decision between the Helvidian and Epiphanian views (and so my version of the family tree leaves this open).”
Excerpt from: The Relatives of Jesus
“The brothers of Jesus were evidently known as ‘the brothers of the Lord’ in early Christian circles (Gal. 1:19; 1 Cor. 9:5), but since the term ‘brother’ by no means necessarily refers to a full blood-brother, the question of their precise relationship to Jesus, along with that of Jesus’ sisters, arises. Since at least the fourth century this issue has been much debated, mainly because of its implications for the traditional doctrine of the perpetual virginity of Mary. The three major views have come to be known by the names of their fourth-century proponents: Helvidius, Epiphanius and Jerome. The Helvidian view, which probably most modern exegetes, even including some Roman Catholic scholars, hold, is that the brothers were sons of Joseph and Mary, born after Jesus. The Epiphanian view, which is the traditional view in the Eastern Orthodox churches, is that they were sons of Joseph by a marriage prior to his marriage to Mary, and so were older than Jesus. The Hieronymian view, which through Jerome’s influence became the traditional western Catholic view, is that they were first cousins of Jesus…Although the Hieronymian view still has its advocates, it must be said to be the least probable. The Greek word for ‘brother’ can be used for relationships more distant than the modern English ‘brother’. However, the brothers of Jesus are invariably called his brothers in early Christian literature (both within and outside the NT). If they were actually cousins, we should expect that this relationship would be specified more exactly on at least some occasions. In fact, the second-century writer Hegesippus, who calls James and Jude ‘brothers of the Lord’, calls Simeon the son of Clopas the ‘cousin of the Lord’, evidently distinguishing the two relationships. But if the Hieronymian view is improbable, it is not easy to decide between the other two views. On the Epiphanian view, the brothers of Jesus would have been his adoptive brothers (assuming the virginal conception of Jesus as historical fact). In that case, we should not expect them to be called anything except ‘brothers’. No NT text offers any further real evidence on this point, but the idea that the brothers and sisters of Jesus were children of Joseph by a previous marriage is found in three second-century Christian works (the Protevangelium of James, the Infancy Gospel of Thomas and the Gospel of Peter), which probably all derive from Syria. It looks as though this was an early second-century Syrian Christian tradition. Reliable tradition about prominent early Christian leaders like the Lord’s brothers could still have been available at this time and place. It is true that the Protevangelium of James implies the perpetual virginity of Mary, and so it is possible that reflection on the idea of the virginity of Mary led to the conclusion that Jesus’ brothers and sisters could not be her children. On the other hand, it is also possible that the notion of the perpetual virginity arose only because Mary was already known not to have been the mother of Jesus’ brothers and sisters. The historical evidence is not sufficient for a firm decision between the Helvidian and Epiphanian views (and so my version of the family tree leaves this open). In any case, we can be sure that the brothers and sisters of Jesus belonged, with him, to the family household of Joseph and Mary in Nazareth. The Gospel traditions regularly refer to Jesus’ brothers in company with his mother.”
 “”The Relatives of Jesus” by
Richard Bauckham,” http://www.biblicalstudies.org.uk/article_relatives_bauckham.html#6 (accessed January 10, 2008).
 For a recent argument, by a Roman Catholic NT scholar, that this is the most probable implication of the NT evidence, see J.P. Meier, A Marginal Jew: Rethinking the Historical Jesus, Vol. 1 (New York: Doubleday, 1991), pp. 316-332; and ‘The Brothers and Sisters of Jesus in Ecumenical Perspective’, Catholic Biblical Quarterly 54 (1992),1-28. For a critique of Meier’s arguments, see R. Bauckham, ‘The Brothers and Sisters of Jesus: An Epiphanian Response to John P. Meier’, Catholic Biblical Quarterly 56 (1994), 686-700.
 Quoted in Eusebius, Hist. Eccl. 2.23.4; 3.11; 3.20.1; 4.22.4.
 Protevangelium of James 9:2; 17:1-2; 18:1; Infancy Gospel of Thomas 16:1-2; Gospel of Peter, according to Origen, In Matt. 10.17.
 Mt. 12:46-47; 13:55; Mk. 3:31-32; 6:3; Lk. 8:19-20; Jn. 2:12; Acts 1:14; Gospel of the Nazarenes frag. 2.