Binary patching rust hot-reloading, sub-second rebuilds, independent server/client hot-reload (#3797)

Enable runtime rust hot patching. * roll back wry to 0.45 * cleanup rustc * wip * wip: fix client/server * get the subsecond cli thing working again * back to list of builds but with accessors * migrate some methods to workspace, clean up impl a bit * pass mode through build context * use a build context, similar to cargo/rustc * merge BuildUpdate and HandleUpdate into BuilderUpdate * Move more resolution of args into runner * migrate some methods to runner * hoist out fullstack * fix build request resolver * yay request is cleaned up * fixup resolution a bit * spawn off the build! * re-wire android autodetection * re-wire android autodetection * wire back up a few more things * re-wire tooling verification * I think it's mostly in the right condition for regular app * add depinfo parser * yay okay works with regular apps again * full rebuilds are back * rewire file wtcher * wire patch! * yayyyyyy patching works and is really fast * yay android works * clean up stdout piping * yayyy memap on android so we don't need root * create a static wasm jump table * wip: global offset table for wasm * wip... thinking about making the patches relocatable * wip: customized relocation * YESSSSSS RELOCATABLE WASM * clean up impl a bit * lil bit more cleanup * lil bit more * lil bit more cleanup * lil bit more cleanup, support 32 bit platforms * sick, wasm is completely pic * hmmmm not quite working yet * woooo, patches loading from rust+wasm * integrated into the cli 😎 * condense a bit * cleaned up request a bit * bust fingerprints * Make the file diffing logic a bit more sensible * still working through a few issues, server launches but is acting weird * fix merge conflict * more merge issues * remove stuff we don't want anymore * revert name change * wip: server not working anymore :( * split apart "dioxus fullstack" into "dioxus server" and "dioxus web" * fixup a few more compile errors * grrr opening server * simultaneous frontend and backend patching * use rustc wrapper * wip: passing assets to jumptable * migrate project-like examples * patchy patchy server fns! * rollback some random changes * unwind the js files * rollback hash * no need for patch * more cleanups * lil bit more cleanup, remove some old cruft * allow patching when wasm webpage is closed * tiny bit more robust * lil bit of clean up * clean up ws code * bit more clean ups * condense * undo file moves * move back other project * move other project * migrate out harness and janky CLI * fix compile * anonymize some files, heavily document others * lots more documentation, for posterity! * ton more docs * clean up the cli a bit more * more ws cleanup * more cleanup * migrate build id to cli-config * add command file handling * random small cleanups * wip.... * wip.... * fix: use workspace for krate discovery * fix panic logging on serve by placing it *after* the logs * reorder logging to be sensible on success * bring back swc * ws cruft * small patches to relocs in loader * bump krates * hoist main funcs into the ifunc table, fixing wasm * fix tui output for long lines * add more linker logging * wow, incredible, we build fat binaries, holy heckkkkk * fix args for frontend/backend * small cleanups * properly send along build IDs to clients * fix workspace compiles * clean up jump table work a bit * clean up logging a bit * open existing * fixup some commands * open existing browser integration * wire up the new dx serve client/server syntax * fixup `dx run` command * bring back some old functionality * fix serverfn feature sets * remove toast animation and simplify its code a bit * less intrusive toast * dont change build status when patching on web * add proper cache busting to vcomponent * clean up the patch file a bit * more lints/checks cleaned up * go back to TargetArgs but a BuildTargets resolved struct * clean up more nits * use an atomicptr for jumptable * fix interaction with initial web load + suspense * don't run asset system on wasm/bindgen js in dev * reduce blast radius of PR * use profile to determine if we're in release * cleanup prod html template * cleanup profiles * fix feature resolution of fullstack * if fullstack is explicitly false, don't use it * light cleanups * drop carg config2 * pass along linker args properly * make workspace examples compile with dx again * fewer unwraps and better error loggign * bit more error handlign * small cleanup * drive-by cleanups * use queries instead of initialize for aslr stuff * fix hotpatch with higher opt level * fix aslr bug from multiple clients * fix merge conflict * fix typos * clippy * fix miscompile * fix doctest * properly rollback wry to 0.45 * fix markdown path issues, other test issues * fix test in router * fix release mode test * fix some more tests, clean up a few more items * use fnptr instead of typeid * clean up memozation via subsecond in core * use "main" as sentinel * fix imports and re-exports * get off __rust_alloc for aslr offset detection * wip * wip... fixing unoptimized web * hmmmm mmmm * close,ish, still missing the wasmbindgen dynaimc imports * aha! full wasm works now - disable externref when compilng patch modules - manually link wasm instrinsics - displace the __wbindgen_placeholder__ table with the synthesized wbg imports what an awful pain. There might be a few items missing, so will test against the docsite shortly * remove failing test * hmmmm things aren't perfect * from what I can tell, relocatable wasm just doesn't work with wasm-bindgen sad https://github.com/rustwasm/wasm-bindgen/issues/1420 * AI IS LITERALLY THE WORST * preserve wasm snippets across compiles * IT HOTPATCHES THE DOCSITE, WE ARE LIVING IN THE FUTURE * remove the macos cfg-out * fix: need to allocate 1 more page * fix url issue * properly span props * properly fill in extra symbols * light cleanups and good docs * delete cruft, better docs * More cleanups and simplifications * clippy, typos * remove harnesses * wasm-bindgen error logging * clean up deps * remove cruft * proper priv/pub * use ifuncs for all env imports * cruft * use par_iter, better output * implement a cache for the base module * delete custom sections * cache the main native module too * add par iter in a few more places * clippy, map ident * better logging and status bars * wip: some small windows support * wip more windowss * fix weird regression * implement windows jump table * small fixes for windows * better windows support * windows support pt2 * rust lld is not msvc * whole archive the rlibs * ignore windows and sys crates * logging * ohhhhhh, pop out dlls * dedupe * thANK GOD FAT LINKING WORKS ON WINDOWS * hoist caching * implement patch based on cache * fix location for pdb * pass data symbols, correct object * add all symbol for windows * add log * treat none rva as undefined * whoops * handle windows calling convention * dont use relocated arm64 stub windows * not all are text? * WINDOWS FUNCTION PATCHING WORKS but issue with statics? * use our own entry * handle windows imp symbols * add log * jump to the real symbol not the layout * whoops * hotpatching windows works completely holy heck yay * drop linker args crate * disable ssg playwright for now * light cleanups * some cleanups / preps: - bump to serverfn 0.8.2 - fix playwright test - fix clippy * use fix from #4042 * bring in other fix for liveview * hoist some configuration * use __aslr_reference instead of aslr_reference * add better errors to patcherror * fix android * panic if patching fails * clean up tomls * wip: fixing wasm compile for server * fullstack wasm again * fix cfg * fixup feature detection * fix issue with features * fix playwright! * add barebones template test harness * fix compatibility between the two hotreload engines * fix preldue of server * fix check for wasm-opt * small fixes
2025-05-06 23:08:59 -07:00
1 parent 71d85e6dd5
commit d976ca184a
@@ -0,0 +1,257 @@
+use itertools::Itertools;
+use std::{path::PathBuf, sync::Arc};
+use target_lexicon::{
+    Aarch64Architecture, Architecture, ArmArchitecture, Environment, OperatingSystem, Triple,
+    X86_32Architecture,
+};
+use tokio::process::Command;
+
+/// The tools for Android (ndk, sdk, etc)
+///
+/// https://gist.github.com/Pulimet/5013acf2cd5b28e55036c82c91bd56d8?permalink_comment_id=3678614
+#[derive(Debug, Clone)]
+pub(crate) struct AndroidTools {
+    pub(crate) ndk: PathBuf,
+    pub(crate) adb: PathBuf,
+    pub(crate) java_home: Option<PathBuf>,
+}
+
+pub fn get_android_tools() -> Option<Arc<AndroidTools>> {
+    // We check for SDK first since users might install Android Studio and then install the SDK
+    // After that they might install the NDK, so the SDK drives the source of truth.
+    let sdk = var_or_debug("ANDROID_SDK_ROOT")
+        .or_else(|| var_or_debug("ANDROID_SDK"))
+        .or_else(|| var_or_debug("ANDROID_HOME"));
+
+    // Check the ndk. We look for users's overrides first and then look into the SDK.
+    // Sometimes users set only the NDK (especially if they're somewhat advanced) so we need to look for it manually
+    //
+    // Might look like this, typically under "sdk":
+    // "/Users/jonkelley/Library/Android/sdk/ndk/25.2.9519653/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android24-clang"
+    let ndk = var_or_debug("NDK_HOME")
+        .or_else(|| var_or_debug("ANDROID_NDK_HOME"))
+        .or_else(|| {
+            // Look for the most recent NDK in the event the user has installed multiple NDK
+            // Eventually we might need to drive this from Dioxus.toml
+            let sdk = sdk.as_ref()?;
+            let ndk_dir = sdk.join("ndk").read_dir().ok()?;
+            ndk_dir
+                .flatten()
+                .map(|dir| (dir.file_name(), dir.path()))
+                .sorted()
+                .next_back()
+                .map(|(_, path)| path.to_path_buf())
+        })?;
+
+    // Look for ADB in the SDK. If it's not there we'll use `adb` from the PATH
+    let adb = sdk
+        .as_ref()
+        .and_then(|sdk| {
+            let tools = sdk.join("platform-tools");
+            if tools.join("adb").exists() {
+                return Some(tools.join("adb"));
+            }
+            if tools.join("adb.exe").exists() {
+                return Some(tools.join("adb.exe"));
+            }
+            None
+        })
+        .unwrap_or_else(|| PathBuf::from("adb"));
+
+    // https://stackoverflow.com/questions/71381050/java-home-is-set-to-an-invalid-directory-android-studio-flutter
+    // always respect the user's JAVA_HOME env var above all other options
+    //
+    // we only attempt autodetection if java_home is not set
+    //
+    // this is a better fallback than falling onto the users' system java home since many users might
+    // not even know which java that is - they just know they have android studio installed
+    let java_home = std::env::var_os("JAVA_HOME")
+        .map(PathBuf::from)
+        .or_else(|| {
+            // Attempt to autodetect java home from the android studio path or jdk path on macos
+            #[cfg(target_os = "macos")]
+            {
+                let jbr_home =
+                    PathBuf::from("/Applications/Android Studio.app/Contents/jbr/Contents/Home/");
+                if jbr_home.exists() {
+                    return Some(jbr_home);
+                }
+
+                let jre_home =
+                    PathBuf::from("/Applications/Android Studio.app/Contents/jre/Contents/Home");
+                if jre_home.exists() {
+                    return Some(jre_home);
+                }
+
+                let jdk_home =
+                    PathBuf::from("/Library/Java/JavaVirtualMachines/openjdk.jdk/Contents/Home/");
+                if jdk_home.exists() {
+                    return Some(jdk_home);
+                }
+            }
+
+            #[cfg(target_os = "windows")]
+            {
+                let jbr_home = PathBuf::from("C:\\Program Files\\Android\\Android Studio\\jbr");
+                if jbr_home.exists() {
+                    return Some(jbr_home);
+                }
+            }
+
+            // todo(jon): how do we detect java home on linux?
+            #[cfg(target_os = "linux")]
+            {
+                let jbr_home = PathBuf::from("/usr/lib/jvm/java-11-openjdk-amd64");
+                if jbr_home.exists() {
+                    return Some(jbr_home);
+                }
+            }
+
+            None
+        });
+
+    Some(Arc::new(AndroidTools {
+        ndk,
+        adb,
+        java_home,
+    }))
+}
+
+impl AndroidTools {
+    pub(crate) fn android_tools_dir(&self) -> PathBuf {
+        let prebuilt = self.ndk.join("toolchains").join("llvm").join("prebuilt");
+
+        if cfg!(target_os = "macos") {
+            // for whatever reason, even on aarch64 macos, the linker is under darwin-x86_64
+            return prebuilt.join("darwin-x86_64").join("bin");
+        }
+
+        if cfg!(target_os = "linux") {
+            return prebuilt.join("linux-x86_64").join("bin");
+        }
+
+        if cfg!(target_os = "windows") {
+            return prebuilt.join("windows-x86_64").join("bin");
+        }
+
+        // Otherwise return the first entry in the prebuilt directory
+        prebuilt
+            .read_dir()
+            .expect("Failed to read android toolchains directory")
+            .next()
+            .expect("Failed to find android toolchains directory")
+            .expect("Failed to read android toolchain file")
+            .path()
+    }
+
+    /// Return the location of the clang toolchain for the given target triple.
+    ///
+    /// Note that we use clang:
+    /// "~/Library/Android/sdk/ndk/25.2.9519653/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android24-clang"
+    ///
+    /// But if we needed the linker, we would use:
+    /// "~/Library/Android/sdk/ndk/25.2.9519653/toolchains/llvm/prebuilt/darwin-x86_64/bin/ld"
+    ///
+    /// However, for our purposes, we only go through the cc driver and not the linker directly.
+    pub(crate) fn android_cc(&self, triple: &Triple) -> PathBuf {
+        let suffix = if cfg!(target_os = "windows") {
+            ".cmd"
+        } else {
+            ""
+        };
+
+        self.android_tools_dir().join(format!(
+            "{}{}-clang{}",
+            triple,
+            self.min_sdk_version(),
+            suffix
+        ))
+    }
+
+    // todo(jon): this should be configurable
+    pub(crate) fn min_sdk_version(&self) -> u32 {
+        24
+    }
+
+    pub(crate) fn ar_path(&self) -> PathBuf {
+        self.android_tools_dir().join("llvm-ar")
+    }
+
+    pub(crate) fn target_cc(&self) -> PathBuf {
+        self.android_tools_dir().join("clang")
+    }
+
+    pub(crate) fn target_cxx(&self) -> PathBuf {
+        self.android_tools_dir().join("clang++")
+    }
+
+    pub(crate) fn java_home(&self) -> Option<PathBuf> {
+        self.java_home.clone()
+    }
+
+    pub(crate) fn android_jnilib(triple: &Triple) -> &'static str {
+        use target_lexicon::Architecture;
+        match triple.architecture {
+            Architecture::Arm(_) => "armeabi-v7a",
+            Architecture::Aarch64(_) => "arm64-v8a",
+            Architecture::X86_32(_) => "x86",
+            Architecture::X86_64 => "x86_64",
+            _ => unimplemented!("Unsupported architecture"),
+        }
+    }
+
+    pub(crate) async fn autodetect_android_device_triple(&self) -> Triple {
+        // Use the host's triple and then convert field by field
+        // ie, the "best" emulator for an m1 mac would be: "aarch64-linux-android"
+        //  - We assume android is always "linux"
+        //  - We try to match the architecture unless otherwise specified. This is because
+        //    emulators that match the host arch are usually faster.
+        let mut triple = "aarch64-linux-android".parse::<Triple>().unwrap();
+        triple.operating_system = OperatingSystem::Linux;
+        triple.environment = Environment::Android;
+        triple.architecture = target_lexicon::HOST.architecture;
+
+        // TODO: Wire this up with --device flag. (add `-s serial`` flag before `shell` arg)
+        let output = Command::new(&self.adb)
+            .arg("shell")
+            .arg("uname")
+            .arg("-m")
+            .output()
+            .await
+            .map(|out| String::from_utf8(out.stdout));
+
+        match output {
+            Ok(Ok(out)) => match out.trim() {
+                "armv7l" => triple.architecture = Architecture::Arm(ArmArchitecture::Arm),
+                "aarch64" => {
+                    triple.architecture = Architecture::Aarch64(Aarch64Architecture::Aarch64)
+                }
+                "i386" => triple.architecture = Architecture::X86_32(X86_32Architecture::I386),
+                "x86_64" => {
+                    triple.architecture = Architecture::X86_64;
+                }
+                other => {
+                    tracing::warn!("Unknown architecture from adb: {other}");
+                }
+            },
+            Ok(Err(err)) => {
+                tracing::debug!("Failed to parse adb output: {err}");
+            }
+            Err(err) => {
+                tracing::debug!("ADB command failed: {:?}", err);
+            }
+        };
+
+        triple
+    }
+}
+
+fn var_or_debug(name: &str) -> Option<PathBuf> {
+    use std::env::var;
+    use tracing::debug;
+
+    var(name)
+        .inspect_err(|_| debug!("{name} not set"))
+        .ok()
+        .map(PathBuf::from)
+}