As noted above, by the year 117 CE under Trajan the Empire reached its greatest size. It encompassed most of England across to Germany and Romania, all of North Africa from present-day Morocco, and extended to the borders of the Parthian Empire. Beyond these borders were “barbarians” of various kinds; as far as the Romans were concerned there were no civilized people outside of their borders except the Persians. Trajan's successor, the emperor Hadrian, built an enormous series of fortifications to consolidate power on the frontiers - these were eventually (by the third century CE) known as the limes, permanent garrisons and fortresses that were meant to serve as barriers to prevent "barbarian" incursions. Some of these survive to the present, including Hadrian's Wall in northern England. While fleets patrolled the rivers and oceans, these garrisons controlled access to the empire.
As far as the Romans were concerned, there were only two things beyond those borders: to the north and northeast, endless tracts of inhospitable land and semi-human barbarians like the "Germans," and to the east, the only other civilization Rome was prepared to recognize: the Persians, ruled first by the Parthian clan and then the Sassanids. For the rest of the Roman Imperial period, Rome and Persia periodically engaged in both raiding and full-scale warfare, with neither side proving capable of conclusively defeating the other.
The closest Rome came to defeating the Persians was under Trajan when he managed to conquer Armenia and parts of Mesopotamia, but after his death Rome swiftly abandoned those territories. Even as they fought, however, Persia and Rome still traded, and Rome also adopted various Persian technologies and military tactics (for example, Rome adopted irrigation techniques from Persia, and Persia adopted engineering techniques from Rome). Persia had the best heavy cavalry in the world, and Rome learned to add heavy cavalry units to its legions by the fourth century CE.
Far beyond Persia was the Chinese Empire, already thousands of years old. China and Rome never established formal diplomatic ties, although the leaders of both empires knew of one another. During the entire period of Roman Imperial power, only China could produce silk, which was highly coveted in Rome. Shipments of silk moved along the aptly-named Silk Road across Central Asia, directly linking the two most powerful empires in the world at the time.
In addition, a major navigational breakthrough occurred during the time of Augustus, when the Romans learned to navigate the Indian Ocean using the Monsoon winds to reach western India. There, they could trade for Chinese silk at much better prices. This journey was hugely risky, but if a Roman merchant could pull it off and return to Rome with a cargo hold full of silk, he would earn fully 100 times his investment as profit. Along with spices (especially pepper), the trade for silk eventually drained enormous amounts of gold from Rome, something that added up to a serious economic liability over the hundreds of years of exchange.
The most important, and threatening, border for Rome was to its north, on the eastern and northern banks of the Rhine and Danube rivers. The region the Romans called Germania was an enormous stretch of heavily forested land, which was cold, wet, and uninviting from the Roman perspective. The “Germans” were a hugely diverse group of tribes practicing feudal law, the system of law in which offenses were met with clan-based violent retribution or blood payments. For hundreds of years there were complex relationships between various tribes and the Roman empire in which the Romans both fought with and, increasingly, hired German tribes to serve as mercenaries. Eventually, some of the Germanic tribes were allowed to settle along the Roman borders in return for payments of tribute to Rome.
The two major rivers, the Rhine and the Danube, were the key dividing lines to the north of Rome, with Roman legions manning permanent fortifications there. As far as the Romans were concerned, even if they were able to militarily they did not want to conquer German territory. The Romans tended to regard the Germans as being semi-human at best, incapable of understanding true civilization. Some Romans did admire their bravery and codes of honor - the same Tacitus who provides much of the information on the early emperors contrasted the supposed weakness and dissolution of his contemporary Romans with the rough virtue of the Germans. That being noted, most Romans believed that the Celts, conquered by Caesar centuries earlier, were able to learn and assimilate to Roman culture, but the Germans, supposedly, were not. Likewise, Germania was assumed to be too cold, too wet, and too infertile to support organized farming and settlement. Thus, the role of the limes was to hold the Germans back rather than to stage new wars of conquest. For about three hundred years, they did just that, until the borders started breaking down by the third century CE.