How hard is factoring?
RSA hides its secret inside the product of two very large primes. Multiplying them to get that number is instant; factoring it back into primes is so slow that, at today's key sizes, the fastest machines on earth would run out of time before the universe did.