the talmud begs to differ:
מֶה הָיָה עוֹשֶׂה, חוֹלֵק לָשׁוֹן שֶׁל זְהוֹרִית, חֶצְיוֹ קָשַׁר בַּסֶּלַע וְחֶצְיוֹ קָשַׁר בֵּין שְׁתֵּי קַרְנָיו, וּדְחָפוֹ לַאֲחוֹרָיו, וְהוּא מִתְגַּלְגֵּל וְיוֹרֵד, וְלֹא הָיָה מַגִּיעַ לַחֲצִי הָהָר עַד שֶׁנַּעֲשָׂה אֵבָרִים אֵבָרִים. בָּא וְיָשַׁב לוֹ תַּחַת סֻכָּה אַחֲרוֹנָה עַד שֶׁתֶּחְשָׁךְ. וּמֵאֵימָתַי מְטַמֵּא בְגָדִים, מִשֶּׁיֵּצֵא חוּץ לְחוֹמַת יְרוּשָׁלַיִם. רַבִּי שִׁמְעוֹן אוֹמֵר, מִשְּׁעַת דְּחִיָּתוֹ לַצּוּק:
What did the one designated to dispatch the goat do there? He divided a strip of crimson into two parts, half of the strip tied to the rock, and half of it tied between the two horns of the goat. And he pushed the goat backward, and it rolls and descends. And it would not reach halfway down the mountain until it was torn limb from limb. The one designated to dispatch the goat came and sat under the roofing of last booth until it grows dark and only then went home. And from what point are the garments of the man rendered impure, as it is stated that he is impure and his clothes requires immersion? From the moment he emerges outside the wall of Jerusalem. Rabbi Shimon says: His clothes are rendered impure only from the moment that he pushes the goat from the cliff.
https://www.sefaria.org/Mishnah_Yoma.6.6?lang=bi