The second caliph, Umar, was murdered by a slave in 644 and the Muslim leaders had to pick the next caliph. They chose an early convert and companion of Muhammad, Uthman. Many members of the Muslim community, however, supported Muhammad's cousin and son in-law Ali, claiming he should be the head of the Ummah, as someone who was part of Muhammad's direct family line. That group was known as the “party” or “faction” of Ali: the Shia of Ali (note that Shia is also frequently spelled “Shi’ite” in English). For Shia Muslims, the central idea was that only descendants of Muhammad should lead the Ummah. The majority of Muslims, known as Sunnis (“traditionalists”), however, argued that any sufficiently righteous and competent leader could be appointed caliph.
While the Shia rejected Uthman’s authority in theory, there was as yet no outright violence between the two factions within the larger Muslim community. In 656 Uthman died, the victim of a short-lived Egyptian rebellion against the Arabs. Ali was elected as the next caliph, seemingly ending the dispute over who should lead the Ummah. Unfortunately for Muslim unity, however, a significant number of Arab leaders disagreed with Ali’s policies and chose to support a rival would-be caliph, a relative of Uthman named Mu’awiya, a member of the Umayyad clan governing Syria. Ali was murdered by a rebel (unrelated to the power struggle over the caliphate) in 661, cementing the Umayyad claim on power, but not the doctrinal dispute between Shia and Sunni.
It was thus under the leadership of caliphs who were not themselves related to Muhammad’s family line that the Arab conquests not only continued, but stabilized in the form of a true empire. The Umayyad clan created the first long-lasting and stable Muslim state: the Umayyad Caliphate. It was centered in Syria and lasted almost 100 years. It oversaw the consolidation of the gains of the Arab armies to date, along with vast new conquests in North Africa and Spain. The Umayyads were capable administrators and skilled generals and the majority of Muslims saw the Umayyad rulers as the legitimate caliphs.
What they could not do, however, was destroy the Shia, despite Ali's death. Shia Muslims, representing about 10% of the population of the Ummah (then and now), viewed the Umayyad government as fundamentally illegitimate, rejecting the very idea of a caliphate and arguing instead that the faithful should be led by an Imam: a direct biological and spiritual descendant of Muhammad’s family. When Ali’s son Hussein, then the leader of the Shia and a grandson of Muhammad himself, was killed by the Umayyads in 680, the permanent breach between Sunni and Shia was cemented.
By 700 CE, the Umayyads had conquered all of North Africa as far as the Atlantic. Then, in 711, they invaded Spain and smashed the Visigothic kingdom, definitively ending Arian Christianity across both North Africa and Spain. They were finally stopped in 732 by a Frankish army led by the Frankish lord Charles Martel at the Battle of Poitiers; this marked the end of the Arab conquests in Europe. Likewise, despite conquering large amounts of Byzantine territory, Constantinople itself withstood a huge siege in 718 and Byzantine forces then pushed back Arab forces in Anatolia.
In Africa, Umayyad armies also attacked Nubia, still one of the richest kingdoms in the region, but were unable to defeat it. For the first time, the caliphate signed a peace treaty with a non-Muslim state; this was an important precedent because it established the idea that a Muslim state could acknowledge the political legitimacy of a non-Muslim one. Afterwards, the Umayyad Caliphate came to deal with non-Muslim powers primarily in terms of normal diplomacy rather than through the lens of holy war.
In 751, Arab forces went so far as to defeat a Chinese army in Central Asia outside of the caravan city of Samarkand (they fought an army of the Tang dynasty, which had been expanding along the Silk Road). The last Umayyad caliph had been murdered shortly before this conflict, however, and the Muslim forces thus had little reason to continue their expansion. This battle marked the furthest extent of the core Muslim-ruled territories. For several centuries to follow, the Muslim world thus consisted of the Middle East, North Africa, and Spain.