jqGrid don't provide currently any simple way to replace the call of editing to another method or to include custom button. What you can do alternatively is "subclassing" $.fn.fmatter.rowactions function like I describe it in the answer. The $.fn.fmatter.rowactions function will be called on click on any from action buttons. So you can test whether the act parameter is "edit" or not. In case of non "edit" button you can forward the call to original $.fn.fmatter.rowactions function. In case of "edit" button you can do any your custom action instead.
UPDATED: The exact implementation depends a little from the version of jqGrid which you use because parameters and the value of this of the function $.fn.fmatter.rowactions are different for different versions of jqGrid. I created for you the demo which uses free jqGrid 4.8 (see readme and wiki). I used the following code for subclassing
var orgRowActions = $.fn.fmatter.rowactions;
$.fn.fmatter.rowactions = function (e, act) {
var $grid = $(this).closest("table.ui-jqgrid-btable"),
rowid = $(this).closest("tr.jqgrow").attr("id");
if (act === "edit") {
$grid.jqGrid("viewGridRow", rowid);
return false;
}
return orgRowActions.call(this, e, act);
}
As the result the "Edit" button starts "View" instead of edit form.
I plan to include more customization possibilities in the next version of free jqGrid. So one will be able to create custom inline icon without the tricks with subclassing.